Documentation for the simeon CLI tool

CLI Guide

Documentation for users of the simeon CLI tool.

• simeon
• Installing from pypi
• Installing with git clone and pip
• Using Docker
• Developing
• Setups and configurations
• List files
• Download and split files
• Make secondary/aggregated tables
• Notes:

Python API Guide

Documentation for contributing to the package, and for those wishing to add simeon to their python dependencies.

• Components of the download package
  • AWS module
    • S3Blob
    • get_file_date()
    • make_s3_bucket()
  • Email opt-in module
    • compress_email_files()
    • parse_date()
    • process_email_file()
  • Tracking logs module
    • batch_split_tracking_logs()
    • process_line()
    • split_tracking_log()
  • SQL data files module
    • batch_decrypt_files()
    • force_delete_files()
    • process_sql_archive()
    • unpacker()
  • Utilities module for the download package
    • check_for_funny_keys()
    • decrypt_files()
    • drop_empties()
    • format_sql_filename()
    • get_course_id()
    • get_file_date()
    • get_module_id()
    • get_sql_course_id()
    • is_float()
    • make_file_handle()
    • make_tracklog_path()
    • move_field_to_mongoid()
    • move_unknown_fields_to_agent()
    • parse_mongo_tstamp()
    • rephrase_record()
    • stringify_dict()
• Components of the upload package
  • GCP module
    • BigqueryClient
    • GCSClient
  • Utilities module for the upload package
    • course_to_bq_dataset()
    • course_to_gcs_folder()
    • dict_to_schema_field()
    • get_bq_schema()
    • local_to_bq_table()
    • local_to_gcs_path()
    • make_bq_load_config()
    • make_bq_query_config()
    • sqlify_bq_field()
• Components of the report package
  • Utilities module for the report package
    • check_record_schema()
    • course_from_block()
    • drop_extra_keys()
    • extract_table_query()
    • get_has_solution()
    • get_problem_nitems()
    • get_youtube_id()
    • make_course_axis()
    • make_forum_table()
    • make_grades_persistent()
    • make_grading_policy()
    • make_problem_analysis()
    • make_roles_table()
    • make_sql_tables_par()
    • make_sql_tables_seq()
    • make_student_module()
    • make_table_from_sql()
    • make_tables_from_sql()
    • make_tables_from_sql_par()
    • make_user_info_combo()
    • module_from_block()
    • process_course_structure()
    • wait_for_bq_job_ids()
    • wait_for_bq_jobs()
• Components of the exceptions package
  • Exceptions module
    • AWSException
    • BadSQLFileException
    • BigQueryNameException
    • BlobDownloadError
    • DecryptionError
    • EarlyExitError
    • LoadJobException
    • MissingFileException
    • MissingQueryFileException
    • MissingSchemaException
    • SQLQueryException
    • SchemaMismatchException
    • SimeonError
    • SplitException