Implemented to improve the overall quality of crime data collected by law enforcement, NIBRS captures details on each single crime incident—as well as on separate offenses within the same incident—including information on victims, known offenders, relationships between victims and offenders, arrestees, and property involved in crimes.
The FBI has maintained both the Summary Reporting System and NIBRS for decades, but contribution to NIBRS the UCR data will now be required for all states. The FBI has declared NIBRS a top priority at the recommendation of the nation’s leading law enforcement organizations. The FBI has established the goal of transitioning to a NIBRS-only crime statistics collection system.
The NIBRS dataset is public and data can be downloaded as flat files, or CSV files which include setup files for PostgreSQL. However, to effectively use the data, it should be data-warehoused into a single database with a comprehensive set of tables that contain all longitudinal data.
The script is run from the command line and will populate a PostgreSQL or MySQL database with the FBI The National Incident-Based Reporting System (NIBRS) data. It is recommended to use PostgreSQL since PG provides better performance over the large data-set. The data is downloaded from an Amazon AWS instance run by the FBI. The data files can also be downloaded one at a time from https://crime-data-explorer.fr.cloud.gov/downloads-and-docs.
The script requires Python 3.6 or higher and will not work properly with Python 2.
The usage of the script is outlined below:
There are three steps.
- Install the required database
- Run the parser NIBRSparser.py
1. Install the database
Run the appropriate database creation scripts depending if you intend to store the NIBRS data in PostgreSQL or MySQL. The script will create a user ‘nibrs’ and limit the scope of the user to the nibrs database. However, for PostgreSQL during the parsing process, the ‘nibrs’ user will be a super-user.
If you want to change the default password for the ‘nibrs’ user, edit the appropriate .sql file before running it. Also, some configuration of your database server maybe necessary depending on the settings choose when running the script. For example the ability to bulk insert CSV files are disabled by default in MySQL.
MySQL or MariaDB
3. Check the log files
The script will keep track of processed files in the LOG directory. There are log files for the individual .zip data files (NIBRS_links.log), as well as the .csv files contained in each .zip file (NIBRS_csv.log) and a main log file NIBRS_app.log which keeps track of errors and warnings from the script. If the script crashes for any reason, you can simply start the script again and it will clear any partially processed data and start where it left off. You can set the verbosity of the stdout and NIBRS_app.log logs with the ‘log_level’ and ‘stdout_level’ variables at the top of the main function.
If you have questions about the FBI NIBRS data you can contact: Author, Joseph Lee: firstname.lastname@example.org