Hi!
We are a group of students that has been assigned a project of redesigning a website that creates statistics of whether municipalities and authorities is signed of IPv6 and DNSSEC.
The server is a linux-based machine that runs bash-scripts to check several domains IPv6 and DNSSEC-capabilities, the scripts creates logfiles daily for this purpose.
The problem we have is that we are not sure it is possible to use the current data of the logfiles and import and store it to a database for a more effective storage since now it is cluttered with logfiles and bad structured.
The current filestructure looks like this
/$date$/dns/domainname.txt so each dns-folder can contain hundreds of files.
An example of a log file of the ones we want to find a way to import looks like this:
0.000: siikalatva.fi INFO Begin testing zone siikalatva.fi with version 1.5.0.
0.218: siikalatva.fi INFO Begin testing delegation for siikalatva.fi.
5.506: siikalatva.fi INFO Name servers listed at parent: ns1.kotinet.com,ns2.kotinet.com,ns3.kotinet.com
7.974: siikalatva.fi INFO Name servers listed at child: ns1.kotinet.com,ns2.kotinet.com,ns3.kotinet.com
7.975: siikalatva.fi NOTICE No IPv6 name servers found.
9.421: siikalatva.fi INFO It is possible to build a referral packet for siikalatva.fi that works without EDNS0.
10.535: siikalatva.fi INFO Previously used name servers for siikalatva.fi:
10.535: siikalatva.fi INFO Done testing delegation for siikalatva.fi.
10.535: siikalatva.fi INFO Begin testing name server ns1.kotinet.com.
10.535: siikalatva.fi INFO Begin testing host ns1.kotinet.com.
10.761: siikalatva.fi INFO Begin testing address 212.50.211.242.
18.262: siikalatva.fi INFO Done testing address 212.50.211.242.
To try and explain it more thorougly
9.421: siikalatva.fi INFO It is possible to build a referral packet for siikalatva.fi that works without EDNS0.
can be divided into these parts 9.421: is just statistics the script runs and doesn't need to be stored.
siikalatva.fi is the domain tested
INFO/NOTICE is the status code and can only be INFO or NOTICE
And the string that follows is the data result which is the most important part.
Our main question for this is that is it possible to use these raw data and create a database to import this or do we need to modify the result for it to be possible?
We have a preliminary history-table that looks like this:
Date (timestamp) (Primary key)
Info (varchar 10)
Data (text)
Warning (text)
Error (text)
the Warning and Error-columns are only preliminary.
If you want to see the website in question to get a better idea it is
http://www.dnssecandipv6.se/
Keep in mind that we are a bit novice in this area.
Regards Daniel, Patrik, Stefan and Malin