|Previous Article||Job Master Index||Next Article|
There are 2 ways that you can process a job, one is more manual than the other, but gives you a greater element of control – that is called Job Processes; the second is the Dashboard which allows you to start a job and then leave the Job Master system to queue the work and run it through to completion in its own time.
The Job Process screen allows you to select a file associated with a job and then manually run the file through each stage of processing.
The processing stages are as follows:
- Load File – simply imports the source data into a SQL Server table behind the scenes.
- Address Update – Performs the specified address correction for the job file.
- Initial Data Analysis Report – generates a report detailing the quality of the data file that has been loaded. This is useful for checking that the data has loaded correctly and also for providing a report to your users illustrating to them the overall quality of the data. Note that Client Suppression files need to have been moved to stage 3 before they can successfully be used in an actual client suppression. There’s a specific option in the dashboard to move a file automatically through the first 3 stages (covered under the Dashboard section).
- Dedupe – performs an internal dedupe on a file and removes duplicate records.
- Client Suppressions – removes records from an input data file that have been found in the client suppression files associated with the job (see job settings).
- Industry Suppression – removes records from an input data file that have been found in any of the standard industry suppression files (GAS, TBR, DisConnect etc).
- Mailsort – runs a mail sortaton as specified in job settings on the remaining data (after stages 1-6) and outputs results to the Reports sub-folder associated with a job.
Run a file through the first 3 stages, and you will see that the stages change colour and when complete, various reports become available:
After reaching stage 3, you’ll also note that lower level dedupe options become available for Dedupe and Client Suppressions. This allows the user to tweak some of the matching options (see the matchIT SQL matching documentation for information on these settings):
Run a file through all 7 stages, and have a look in the reports sub folder for a job – you will see a sub folder created with a name like BBSMailSort_[JOB NAME]_1; the generated Mailsort ouput is found here. Congratulations you have completed a processing job using the Job Master system.
This page lists each file registered to a job and allows you to either manually run a process stage, or run all of the available process stages for a file. There is also a special option call ‘Load, PAF and Data Download Rpt’ which will move a selected file through all of those stages in one go; handy for getting client suppression files ready for processing.
If you select Run All, then Job Master will queue the various job stages and run then in order as processing resource becomes available. The Dashboard web page will automatically update every 60 seconds with the latest status for your job allowing you to just leave it to process in the background.
Job Stage Output Files
The following job stages produce interim output files in the reports sub-folder of your job:
- Initial Data Analysis - cleaned_afterdatadownload_[ Job File ID].txt – address verified, normalised format of supplied input data.
- Dedupe – 2 output files are produced. cleaned_deduped_[ Job File ID].txt - address verified, normalised format of supplied input data excluding identified internal duplicate records. cleaned_duplicates_[ Job File ID].txt – list of duplicate records that have been removed.
- Client Suppressions – 2 output files are produced. cleaned_clientsuppresion_duplicates_[ Job File ID].txt - address verified, normalised format of supplied input data excluding identified internal duplicate records and client suppressions. cleaned_clientsuppresion_duplicates_[ Job File ID].txt – list of duplicates that have been found and removed during the client suppression stage.
- Industry Suppressions – 3 output files are produced. suppression_cleaned_[ Job File ID].txt - – address verified, normalised format of supplied input data, minus internal duplicates, client suppressions and industry suppressions. suppression_hits_[ Job File ID].txt – list of records that matched against an industry suppression file. suppression_source_cleaned_[ Job File ID].txt – original source data minus internal duplicates, client suppressions and industry suppression hits.
- Mailsort - BBSMailSort_Example(1)_[CELL ID] – sorted mailsort results including planning report, bag reports etc.
Note that interim output files can be switched off if required, by the setting ‘OUTPUT_INTERIM_RESULTS’ setting to ‘false’ in the appsettings table of the JobMaster database.
Job Stage Database Tables
As well as outputting data to file when processing a job, a new database is created for each job and output tables are created within that database as processing proceeds.
Initial Data Analysis
- InputFile_[Job File ID]__HHGBase__ – address verified, normalised format of supplied input data.
- [Job File ID]_deduped – deduped source data – i.e. internal matches removed.
- [Job File ID]_duplicates – records identified as duplicates and removed.
- [Job File ID]_ClientSuppressions_MatchedIDs – records from InputFile_[Job File ID]__HHGBase__ which matched mapped client suppression files.
- [Job File ID]_ IndSupps_PrioritizedHits – records from InputFile_[Job File ID]__HHGBase__ which matched mapped industry suppression files.
- Cell_[CELL ID]_MailSortation – sorted data following the Mailsort stage.
|Previous Article||Job Master Index||Next Article|