STOCK DATA PROJECT REPORT

For MySQL databases & tables

Akanksha
Report Version 1
 1

Summary

This version 1 report explains how to setup and administer the MySQL database for storing

stocks’ daily price data.

The report will cover the following steps:

 Setting up 2 MySQL instances/servers for Development (DEV) and Production (PROD)

 The ERD for tables in DEV and PROD

 Setting up 2 databases for DEV and PROD

 Setting up the tables for storing daily price data in DEV and PROD

 Updating the tables in DEV and PROD with daily price data for the stocks

 Generating reports from the daily price data stored in PROD

 2

Contents

Summary ................................................................................................................................................. 1

MySQL Servers for Development and Production .................................................................................. 3

MySQL Databases for Development and Production ............................................................................. 4

Entity-Relationship-Diagram ................................................................................................................... 5

MySQL Tables for Development and Production.................................................................................... 6

MySQL Daily Price Data for Development and Production .................................................................... 6

MySQL Reports for Production ............................................................................................................... 7

Appendix ................................................................................................................................................. 8
 3

MySQL Servers for Development and Production

We will create 2 MySQL server instances on our local machine for PROD and DEV each. This

will ensure restricted access to PROD environment and safeguard against errors, and ensure

DEV scripts are not run in its schema accidentally. For the first server instance download

MySQL from your browser and follow the installation steps as directed. For the second

instance, the steps for Windows 10 are as follows:

Copy
 C:\ProgramData\MySQL\MySQL Server 8.0\my.ini
1.
to
 C:\ProgramData\MySQL\MySQL Server 8.0\my1.ini
Open my1.ini and modify:
 Port = 3307(under Client and Server Section)
2.
 Datadir = C:/ProgramData/MySQL/MySQL Server 8.0/Data1
 report_port = 3307
Copy
 C:\ProgramData\MySQL\MySQL Server 8.0\Data
3.
to
 C:\ProgramData\MySQL\MySQL Server 8.0\Data1
Run on cmd prompt: (With Administrator privileges if necessary)

4. C:\Program Files\MySQL\MySQL Server 8.0\bin>mysqld --install MySQL80-1 --defaults-

file="C:\ProgramData\MySQL\MySQL Server 8.0\my1.ini"
If all went well, you will see:
5.
 Service successfully installed

6. Type services.msc, find the service name MySQL80-1, right-click on it and click Start.

If all went well, you will see the Status change to Running.

7. If it did not go well, open xxx.err file found in C:\ProgramData\MySQL\MySQL Server

8.0\Data1 to check why.
 4

MySQL Databases for Development and Production

Now that we have two MySQL server instances [(DEV) MySQL80 | (PROD) MySQL80-1], we

will create the respective databases on the servers through MySQL Workbench.

* Stop the MySQL80 DEV service from Services.msc on your Windows machine (preferably)

when creating PROD databases. This will reduce load on your hardware and also protect PROD

data completely.

1. Right click in the MySQL Workbench Navigator and click Create Schema.

2. Name the schema as DEV or PROD according to your workbench instance.

Make sure you are logged in as root (Users and Privileges) in both databases. This is
3.
important for the future steps of the setup.

In the schema list on both workbenches you will see DEV and PROD schemas respectively
4.
if everything runs correctly.
 5

Entity-Relationship-Diagram

For the DEV and PROD databases we have created a Strong Entity table called Ticker_List

which will be a directory of all the stocks we will be collecting data on. The id from this table

will form a Foreign Key reference to the other Strong Entity table Daily_Ticker_Price table

column ticker_id, so we don’t have to insert the name of the stock repetitively. The

relationship between Ticker_List and Ticker_Price_History is ‘One is to One’. The relationship

from Ticker_List to Daily_Ticker_Price is ‘One is to Many’. The day the stock data was created

will be stored under created_date column for all three tables, with a Foreign Key reference

from the Ticker_List table. The Ticker_Price_History table will also have a Foreign Key

reference for the ticker_id column. The last_updated column, which stores the date and time

of the last price change, will be stored in Ticker_Price_History, with a Foreign Key reference

to the Daily_Ticker_Price. So the relationship here is ‘One is to Many’.

*[The Daily_Ticker_Price table will be partitioned by id and ticker_id to be able to query the

table in slices, which will be faster, given the millions of rows that will be stored in this table.

So individual stock data can be queried by just selecting the partition p0, p1, p2…..pn.]
 6

MySQL Tables for Development and Production

For Windows 10, from command prompt or Visual Studio:

 Run the CreateTables_DEV.Py script to create the aforementioned tables (in ERD) in

DEV

If the script runs successfully:

 Run the CreateTables_PROD.Py script to create the tables in PROD

* Stop the MySQL80 DEV service from Services.msc on your Windows machine (preferably)

when running scripts in PROD. This will reduce load on your hardware and also protect PROD

data completely.

MySQL Daily Price Data for Development and Production

For Windows 10, from command prompt or Visual Studio:

 Run the Multiprocessing_CSV_DEV.Py script to insert the CSV files’ data into DEV

tables

If the script runs successfully:

 Run the Multiprocessing_CSV_PROD.Py script to insert the CSV files’ data into PROD

tables

* Stop the MySQL80 DEV service from Services.msc on your Windows machine (preferably)

when running scripts in PROD. This will reduce load on your hardware and also protect PROD

data completely.
 7

MySQL Reports for Production

 Run the Reports_PROD.Py script to generate a report of the data stored in the PROD

tables

 The report will be directly emailed to the GMAIL account of the user (Python’s package

sql_to_html used in the .Py file)

* Stop the MySQL80 DEV service from Services.msc on your Windows machine (preferably)

when running scripts in PROD. This will reduce load on your hardware and also protect PROD

data completely.
 8

Appendix

1. Partitioning Tables

CREATE TABLE DAILY_TICKER_PRICE (

id serial NOT NULL,
stock_id serial NOT NULL,
price_date date NOT NULL,
created_date date NOT NULL,
last_updated date NOT NULL,
open_price decimal(11,4) NOT NULL,
high_price decimal(11,4) NOT NULL,
low_price decimal(11,4) NOT NULL,
close_price decimal(11,4) NOT NULL,
volume decimal(11,4) NOT NULL
)
PARTITION BY stock_id;

2. CreateTables_*.py
This script will create the tables with the primary key, foreign key, indexes and partitioning references

such as above. The id and stock_id will be the primary keys and indexes in the daily_ticker_price table.

3. Multiprocessing_CSV_*.py

This script will insert all the CSV files’ data into the tables. They will drop the INDEXES at the beginning

of the script before beginning the inserts and add them back at the end of the script, before closing

connection to the database (Indexes slow down inserts and updates but fasten select queries).

4. DDL and DML statements

All such statements will be run within TRIGGERS or BEGIN…END blocks such as below to prevent bad

queries from corrupting the tables. The statements will only be committed at the end of the block;

any errors will ‘roll back’ the entire script.

CREATE TRIGGER trigger_name

{BEFORE | AFTER} {INSERT | UPDATE| DELETE }
ON table_name FOR EACH ROW
trigger_body;
[begin_label:] BEGIN
[statement_list]
END [end_label]