Usage Guide¶
This guide explains how to use the tea-data-file-conversion package via the command-line interface (CLI) and programmatically through the processor module.
Command-Line Interface (CLI)¶
The CLI converts a TEA data file into CSV format using a dynamic YAML schema. It reads two input formats:
Fixed-width, where each field occupies a fixed column range.
Delimited (CSV), where fields are identified by a header column name.
The format is inferred from the input file’s extension: a file ending in .csv is read as delimited, and anything else is read as fixed-width. A delimited file must therefore be named with a .csv extension to be parsed correctly — a fixed-width read of delimited content, or vice versa, fails with an error rather than producing silently wrong output.
Display Help¶
To see the available commands and options, run:
tea-data-file-conversion --help
Process a Fixed-Width File¶
To convert a fixed-width file to CSV using a schema, use the following command:
tea-data-file-conversion input_file.txt --output_file output.csv --schema_folder path/to/schema
input_file.txt: Path to the fixed-width input file.--output_file output.csv(optional): Path for the converted CSV file. If omitted,_output.csvis appended to the input filename.--schema_folder path/to/schema(optional): Path to the folder containing YAML schema files.
Process a Delimited (CSV) File¶
To convert a delimited file, pass a .csv input the same way:
tea-data-file-conversion input_file.csv --output_file output.csv --schema_folder path/to/schema
Delimited files are identified differently than fixed-width ones: the test type comes from the input filename (for example, a filename containing ACCOUNTABILITY selects the consolidated_accountability schema) and the school year comes from a YEAR column in the file’s first data row, rather than from a header prefix. The schema selected for the input must use the matching field shape — see the “YAML Schema Format” section below.
Exporting YAML Schema Templates¶
To export default YAML schema templates for customization, run:
tea-data-file-conversion --export_templates --schema_folder path/to/schema_folder
This will copy built-in templates into the specified folder.
Processor Module¶
If you need to integrate file conversion into a Python script, use the processor module.
Processing a File Programmatically¶
You can use the process_file function to process either a fixed-width or a delimited (.csv) file programmatically; it infers the format from the input file’s extension the same way the CLI does:
from tea_data_file_conversion.processor import process_file
# Define file paths
input_file = "data/input_file.txt" # or "data/input_file.csv" for a delimited file
output_file = "data/output_file.csv"
schema_folder = "schemas"
# Process the file
df = process_file(input_file, output_file=output_file, schema_folder=schema_folder)
print("File converted successfully!")
Exporting YAML Schema Templates Programmatically¶
To export the built-in YAML schema templates via a script:
from tea_data_file_conversion.processor import export_templates
export_templates("schemas")
Generating a YAML Schema from a CSV File¶
If you have a CSV defining field positions, you can convert it to a YAML schema:
from tea_data_file_conversion.processor import csv_to_schema_yaml
csv_to_schema_yaml("fields.csv", yaml_output_file="schema.yaml")
YAML Schema Format¶
YAML configuration files define how the input file should be parsed. A schema is either fixed-width or delimited, and its fields must all use the same shape — the two are not mixed within one file.
Fixed-width fields locate data by column position:
fields:
- start: 1
end: 10
output_field: "StudentID"
keep: true
- start: 11
end: 20
output_field: "Score"
keep: false
Delimited fields locate data by the column name in the input file’s header, via source_column in place of start/end:
fields:
- source_column: "STU_ID"
output_field: "StudentID"
keep: true
- source_column: "SCORE"
output_field: "Score"
keep: false
Both shapes support two further optional keys on each field: keep (a boolean; only fields marked true are returned when process_file is called with filter_columns=True) and mapped_field_name (an alternative output name used in place of output_field when filtering).
Schema Validation¶
The validate_yaml_config function ensures a schema follows one of these two structures; an invalid or mixed-shape schema raises an error. Delimited schemas are additionally required to have unique output_field values, since process_delimited_file renames columns by that name and a duplicate would silently fan one source column out into several. Fixed-width schemas are exempt from that particular check because process_fixed_width_file de-duplicates repeated output names itself at runtime.
Error Handling¶
If an error occurs while processing a file, ensure:
The schema file exists at the expected path and is correctly formatted for its format (fixed-width or delimited).
The input file follows the expected format for its extension — fixed-width for anything other than
.csv, delimited for.csv.The schema’s field shape matches the input format: a
.csvinput needs a schema whose fields usesource_column; any other input needs a schema whose fields usestart/end.For delimited files, a
YEARcolumn is present in the data and the filename identifies a known test type (for example, containsTELPASorACCOUNTABILITY).
When using the processor module, wrap function calls in a try/except block:
try:
process_file("input.txt", "output.csv", schema_folder="schemas")
except Exception as e:
print(f"Error: {e}")