Covid19 Reporting

 

Release Notes

0.7.13 (April 4th, 2020)

  • New data for 4/4 loaded from JHU CSSE

0.7.12 (April 3rd 2020)

  • Fixed a bug where if you had ‘treat provinces/states as countries’ set to “no”, that the reset links to show all states/provinces within US/Australia/Canada/etc did not work properly
  • Bumping up the required version of the Canary app, so as to pull in some important additions from that side.
  • Some wording improvements to the form element options
  • New data for 4/3 loaded from JHU CSSE

0.7.11 (April 2nd, 2020)

  • New data for 4/2 loaded from JHU CSSE
  • The experimental “forecast” panel now only appears if one of the statistics that actually does have an ‘exponential’ phase, is selected — ie total deaths, total cases, and total ‘sick’. Conversely if something like Case Fatality Rate is selected, it hides itself.

0.7.10 (April 1st, 2020)

  • New data for 4/1 loaded from JHU CSSE – includes the US data that was missing at 4:50pm pacific

0.7.9 (April 1st, 2020)

  • Added a new y-axis option of ‘case fatality rate (then-current)’. This option switches you to a linear y-axis and disables the logarithmic y-axis. (h/t to Ari for the conversation that indicated this was a bit of a hot button statistic.)
  • Added new y-axis options to allow charting of new deaths per day, as opposed to cumulative deaths. likewise new confirmed cases per day as opposed to cumulative cases per day.
  • Fixed a bug – the shortcut links to reset selected countries/and states now preserve your x-axis normalization options
  • Added a median_age field to the us_states.csv
  • in the underlying SPL there are new fields in the underlying datacube, of median_age by country (or for US States, for that state), also Population and “Urban_Percentage”
  • minor improvements to organize the todos and notes and link to them from the nav
  • New data for 3/31 loaded from JHU CSSE, but this is a very thin daily report and for instance it includes no new points for any US states

0.7.8 (March 31st, 2020)

  • A small change to condense the form elements slightly to help users with smaller monitors.
  • Manually preventing France and the United Kingdom from being split apart into province/state values because they only have a couple small province/state values being broken out and since they are far smaller in population than the respective “most of the country” value, the net effect on the UI is a little confusing. and I’m going to revisit this later to find a better solution.
  • added a new static lookup for countries, that has population, land area, median age, and ‘urban percentage’. This isn’t wired up as anything in the UI yet at all, but it will be soon.
  • (New data for 3/31 loaded from JHU CSSE)

0.7.7 (March 30th, 2020)

  • Edited the ‘default’ reset link to add Belgium and the Netherlands.
  • Fixed a bug in the new US-counties prototype, where the state pulldown was initially selected to “all”
  • Fixed a bug where extracting the county names from jan/feb rows, inadvertently trimmed off some city-level rows. Now the group that captures the county names is optional in the regex.
  • Added back the testing options to the y-axis pulldown (these pull from the covidtracking.com data, and while the machinery has been in the app for some time, I inadvertently removed the options from the y-axis several releases ago)
  • Added a contextual disclaimer about the “PendingTests” statistic, that these numbers are not widely reported so the chart can be misleading.
  • (New data for 3/30 loaded from JHU CSSE)

0.7.6 (March 29th, 2020)

  • Added some higher numbers to the “by number of days since the” selector. Unfortunately it is sometimes useful to normalize the x-axis to these much higher numbers now.
  • Moved the layer of reset links into its own Pattern file.
  • Added a new prototype that allows the same sort of interactivity but specifically for the counties within one particular US State. For all non-US users out there, unfortunately the JHU dataset recently started providing this County-level detail only for US States. For what it’s worth when similar granularity is available for other countries I will certainly try and expose that as additional functionality, as I’ve done here for the US.)
  • (New data for 3/29 loaded from JHU CSSE)

0.7.5 (March 28th, 2020)

  • Added a disclaimer to both “cases over time by country or state” and “recovery by country” to call out that the “Recovered” numbers in the dataset are not broken out by US State. Without this disclaimer it seemed potentially quite alarming. Hopefully the JHU CSSE folks are able to soon break this down by state and I can change this disclaimer or remove it later.
  • (New data for 3/28 loaded from JHU CSSE)

0.7.4 (March 27th, 2020)

  • (New data for 3/27 loaded from JHU CSSE)

0.7.3 (March 27th, 2020)

  • Fixed a misconfiguration of the main view, where the main Country/Province/State pulldown would reset to the default every time you reloaded the page (rather than using the selection preserved in the url)
  • minor fix so the chart doesn’t redispatch when you just change the y-axis type.
  • Expanded the ‘reset to default’ link to also allow you to reset to selecting all the provinces within the few countries that have that level of granularity. This is only a convenience but it saves a lot of time selecting/deselecting provinces and states.
  • Also I have added another static lookup to handle normalization of various Province_State names that have typos or inconsistencies.
  • (New data for 3/26 loaded from JHU CSSE)

0.7.2 (March 26th, 2020)

  • Fixed a minor bug where the explanatory note above the main chart always said the xaxis showed days since the Nth “confirmed case”, even when that option was set to ‘deaths’ instead.

0.7.1 (March 25th, 2020)

  • Posting a quick fix for a regression in the 0.7 build just posted, where ‘recovery-over-time’ tab didn’t load.

0.7 (March 25th, 2020)

  • The app now ships a pair of lookups. These are a snapshot of
    index=covid19 sourcetype=”csse_covid_19_daily_report”
    and
    index=covid19 sourcetype=”states_daily_testing”
    respectively. All searches in the app draw from these static lookups and not from searching indexed data. The app does still ship a snapshot of the daily reports data in the “covid_data” subfolder. However it does not ship enabled data inputs to index those files.

    This is a new solution to the problem we’ve had, where when JHU fixes typos or cleans something up in an old daily report, that the monitor inputs would then reindex part of the data when the user updates to a new version. These nature of this data, and the nature of these updates are such that ‘simply deduping’ the events is not as easy as it normally is in Splunk.

  • I’ve put back the legacy SideviewUtils module into all views. I removed this by mistake thinking it was redundant now. However this is still a load-bearing piece of how we’re ensuring redirects away from legacy advanced-xml URLs to proper canary URLs.
  • (New data loaded from JHU CSSE dataset)

0.6 (March 24th, 2020)

  • Improved the `get_count_of_days_since_nth_case` further, so now it can take three optional args. See comments in macros.conf
  • Added the ability to explode out the countries/provinces and treat them as separate countries. This allows a much more flexible analysis – notably you can compare individual US states or Australian or Canadian provinces to entire countries. The previous tab doing per-state analysis has been deleted.
  • Added controls to allow the user to change from the x-axis being “days since the 100th confirmed case” from 100 to any N, and from “confirmed case” to “death”.
  • Fixed a bug where clicking ‘see raw search syntax’ on a tabular output would dump you out into a column chart
  • Pulled out the primer on logarithmic y-axes into its own page under ‘notes’
  • Upped the required version of Canary to 1.2.3, and added a little message that appears in the top of the analysis view, if the version is too low

0.5.2 (March 23rd, 2020)

  • Fixed a problem in the “COVID-19 dataset” tab where the covidtracking.com data rows were appearing there, but with most columns blank. As this was confusing and served no purpose I filtered them out
  • Fixed a regression in the “COVID-19 dataset” tab where the selection of the province/state element had no effect on the results.
  • Added some text underneath the main tab giving users a primer on exponential curves and how to interpret charts with logarithmic y-axes
  • Since the JHU CSSE data suddenly has a whole other field for US County, but the field name strangely is “Admin2”, I’ve renamed the field to “County” and made it visible in the “COVID-19 dataset” tab.
  • (New data loaded from JHU CSSE dataset)

0.5.1 (March 23rd, 2020)

  • Fixed an unfortunately bug that only happened on 7.3, but that prevented users from being redirected into the app correctly. Instead they were stranded on a page telling them that “TopNav” and “AppNav” modules didnt exist.

0.5 (March 22nd, 2020)

  • Reviewed and made improvements to how we handle days with missing data at the country and province level. Previously the SPL screened out the duplicate records where the “Last Update” referred to a day before the filename the record was in. Now we let those records get indexed, but at search time we override the _time value from the filename (Normally this is a bad practice in Splunk, it’s better to do this with INGEST_EVAL but here it’s quick, its effective and we pull the entire set off disk anyway so there’s not much of a downside.)

0.4.3 (March 21st, 2020)

  • Removing the warning message added in 0.4.2 and the app now simply dedups the data explicitly on search. The data set is so tiny it really doesn’t have an effect.
  • There is a new health checks page, whereby the app ships stanzas in checklist.conf that get run explicitly on this page. Note the checklist.conf stanzas will also run in the Splunk Monitoring Console if any splunk admins are looking there
  • (New data loaded from JHU CSSE dataset)

0.4.2 (March 21st, 2020)

  • JHU recently went back and changed something in the daily data from 2 days ago, which is great. However this comes before I had completed the health check to automatically detect this and bubble up a warning to the user.
    So this release just has a hardcoded warning at the top to always reindex the data after updating to a new version.

0.4.1 (March 20th, 2020)

  • Redoing the default country selection. The short version is “I added Switzerland”.
    The Long version – Previously it was the top 10 total confirmed cases, plus an extra 2 – Hong Kong and Singapore which are low but have extremely interesting curves. New it is the top 10, plus an extra 3 – Japan, Hong Kong and Singapore.

0.4 (March 20th, 2020)

  • the app now ships a snapshot of the per-day testing data for US States from https://covidtracking.com/api/
  • some key numbers from the us testing data can now be chosen as y-axis values on the per-province/state chart, although of course they only work for US states at the moment.
  • In the “by province/state” tab I have marked the “Recovered” yaxis option as “(doesn’t work with recent US data)”, The reason is that since the “Recovered” numbers in the US are no longer associated with individual states, this means we can’t really graph that number in the US split by state. While you might think we can create a fake state called ‘unknown’, we’d ahve to also determine “on what day the unknown state passed 30th confirmed cases” and that’s nonsensical.

0.3.4 (March 20th, 2020)

  • Altered the app so that even if the user fails to install the Sideview Utils app or the Canary app (or both) this app itself will explain that problem to them on the landing page and give them instructions for how to install the missing apps. (Before this version the app would just load a blank page and leave the user stranded.)
  • Tested on Splunk 7.3 and since everything seems fine, I am marking it compatible with 7.3 (prior to this it was marked in SB as Splunk 8.X only)
  • gave the navigation bar a link to the release notes file.

0.3.3 (March 19th, 2020)

  • (New data loaded from JHU CSSE dataset)

0.3.2 (March 19th, 2020)

  • Fixed the problem that was preventing other countries that had more than 1 Province/State past the 30-confirmed-cases mark, from appearing in the ‘cases over time by province/state’ graph. Short Version – you can now see Australia, Canada and Denmark in that visualization
  • Changing the “cases over time by province/state” tab’s y-axis to default to normal ie ‘linear’ because it’s actually more informative that way, at least at this stage.
  • Changing the ‘recovery over time by country’ page to use line charts instead of area charts. I think with area charts it’s slightly easier to misinterpret the numbers as “net-new cases for that day” when they are in fact cumulative numbers.
  • Adding a state/province CheckboxPulldown to the “COVID-19 dataset” tab. Also adding a second Pager control at the bottom of the table for convenience.
  • Fixing an issue that only just appeared in the 3/18/2020 data for the US. There’s a row with Province_State=”US” and Recovered=106 but confirmed is only listed as 1. I am interpreting this to mean the confirmed is reported at a more granular level already, but through some datasource they know there are 106 recovered cases across the country. The change specifically here to this app, is that the “Sick” number would for this one datapoint, go negative and thus throw off numbers for that day.
  • Patching a minor problem in the data where sometimes the United Kingdom’s “United Kingdom” Province is listed as “UK”.
  • moving the log/linear axis control in the province/state tab to be below the other controls whose changes redispatch the search.
  • (New data loaded from JHU CSSE dataset)

0.3.1 (March 18th, 2020)

  • small change to the “recovery by country” tab, to make their time axis all start on the same day. This resolves at least the constant need to refer to the x-axis as you look between charts.

0.3 (March 18th, 2020)

  • New tab added “cases over time by province/state”. This is much like the main visualization showing change over time since the day the country had its 100th total confirmed case, except that you pick an individual country and then it does that analysis for the country’s provinces or states.
  • SPL has been developed to make the US data from the early days more usable. The states are extracted from the county names and so ultimately that county-level data can get automatically counted when analysis is done split or grouped by state.
  • as a part of the previous item, added a small csv lookup that maps 2-letter abbreviations for US states to the full state name.
  • Data Onboarding Notes page has been updated. Also several macros have been renamed and several new macros have been created. Note that macros.conf also contains quite a lot of comments for anyone interested in dissecting how this all works.
  • release notes file created, and backfilled from checkin comments for prior 2 releases.
  • (New data loaded from JHU CSSE dataset)

0.2.4 (March 18th, 2020)

  • New (hidden) prototype of the visualization to split things by Province/State for particular countries.
  • parametrizing the “get_count_of_days*” macro and abandoning the old hardcoded one that did the 100 day threshold.
  • (New data loaded from JHU CSSE dataset)

0.2.3 (March 17th, 2020)

  • changing the main tab name from “confirmed cases over time” to “cases over time by country”. The old name I think made it easier to think the ‘sick cases’ number on the y-axis and the ‘time since the 100th confirmed case’ on the x-axis, were the same number. Might revisit this again but this seemed an easy improvement.
  • Fixed a typo that broke one of the links to the data_onboarding_notes page
  • added the working list of todo’s into the app at the bottom of the data_onboarding_notes page.
  • (New data loaded from JHU CSSE dataset)

0.2.2 (March 17th, 2020)

  • css tweaks.
  • making the “remaining sick patients” yaxis option to “remaining patients sick or testing positive”. In some informal user testing someone assumed incorrectly from the old label that the patients were visibly sick and showing symptoms, whereas the count also includes patients testing positive but asymptomatic.
  • adding the nullValueMode=connect to the “recovery by country” tab.
  • moving the Sick field to a calculated field.
  • manually filtering out the 3/1 data point where only ‘channel islands’ reported data in the UK. with the new nullValueMode=connect, it’s better to just yank the point, let the charting framework connect it, rather than what I was doing prior creating a fake data point (ick). Now the chart looks the same but if you look closely you’ll see there’s no little data point circle there and that’s better.
  • commented the macros.
  • (New data loaded from JHU CSSE dataset)

0.2.1 (March 16th, 2020)

  • Fixing a bug – yaxis was always showing “confirmed cases”. Now it reflects the actual selected yaxis statistic.
  • Fixing the linux/windows problem with that filename extraction.
  • Adding a second data input with windows backslashes, so that the app’s data input can work out of the box on Windows as it does on *nix
  • (New data loaded from JHU CSSE dataset)