A reference implementation for processing the content.log files found at opendata.dwd.de/weather

Last update: Nov 26, 2022

Related tags

Miscellaneous opendata-content.log-tool

Overview

content.log Tool

A reference implementation for processing the content.log files found at opendata.dwd.de/weather.

Example usage:

wildcards.txt ./get_updated_files.py -w wildcards.txt 2022-01-26T03:00 > updated_files.txt">

wget https://opendata.dwd.de/weather/nwp/content.log.bz2
bunzip2 content.log.bz2
echo "./icon-d2/grib/*/t_2m/*" > wildcards.txt
./get_updated_files.py -w wildcards.txt 2022-01-26T03:00 > updated_files.txt

The produced file updated_files.txt will hold all pathnames as given by content.log that are updated since 2022-01-26 03:00:00 UTC according to the file's modification date. Remember that those paths are relative to the directory the content.log.bz2 was in, in this case https://opendata.dwd.de/weather/nwp/.

Also mind that there are multiple servers behind https://opendata.dwd.de which might not be exactly in sync with each other regarding file modification timestamps. Look into the code of get_updated_files.py for a suggestion on how to deal with that.

While this program relies on the file modification timestamp dumped into content.log.bz2, it might be more feasible to process the data reference time that is contained in the filenames instead.

$ ./get_updated_files.py --help
usage: get_updated_files.py [-h] [--content-log CONTENT_LOG_FILE] [--wildcards WILDCARDS_FILE] [--min-delta MIN_DELTA] [--version] UPDATED_SINCE

Filters paths of a DWD Open Data content.log file for entries that have been updated.

positional arguments:
  UPDATED_SINCE         Last time files were checked for updates

optional arguments:
  -h, --help            show this help message and exit
  --content-log CONTENT_LOG_FILE, -f CONTENT_LOG_FILE
                        The decompressed content.log file (default: content.log)
  --wildcards WILDCARDS_FILE, -w WILDCARDS_FILE
                        Filter results by a set of wildcards
  --min-delta MIN_DELTA, -d MIN_DELTA
                        Minimum number of seconds a file needs to be younger than UPDATED_SINCE (default: 60)
  --version             show program's version number and exit

Script de monitoramento das teclas do teclado, salvando todos os dados digitados em um arquivo de log juntamente com os dados de rede.

listenerPython Script de monitoramento das teclas do teclado, salvando todos os dados digitados em um arquivo de log juntamente com os dados de rede.

4 Nov 27, 2022

Python Common things by Problem Fighter Library, (Exception, Debug Log, etc.)

In the name of God, the Most Gracious, the Most Merciful. PF-PY-Common Documentation Install and update using pip: pip install -U xxxx Please find the

3 Jan 15, 2022

A log likelihood fit for extracting neutrino oscillation parameters

A-log-likelihood-fit-for-extracting-neutrino-oscillation-parameters Minimised the negative log-likelihood fit to extract neutrino oscillation paramete

1 Jan 23, 2022

A Python Based Utility for Processing GST-Return JSON Files to Multiple Formats

GSTR 1/2A Utility by Shan.tk Open Source GSTR 1/GSTR 2A JSON to Excel utility based on Python. Useful for Auditors in Verifying GSTR 1 Return Invoices

1 Oct 8, 2022

A python script that changes your desktop background based on current weather and time of the day.

Desktop background wallpaper, based on current weather and time A python script that changes your computer's desktop background based on current weath

1 Nov 16, 2021

A simple weather app.

keather A simple weather app. This is currently not finished. Dependencies: yay -S python-beautifulsoup4 tk

1 Jan 9, 2022

Use a real time weather API to apply wind to your mouse cursor.

wind-cursor Use a real time weather API to apply wind to your mouse cursor. Requirements PyAutoGUI pyowm Usage This program uses the OpenWeatherMap AP

1 Feb 7, 2022

Python package for reference counting native pointers

refcount master: testing: This package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boi

2 Nov 3, 2022

Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters.

Project Faros Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters. The project includes referen

9 Jul 18, 2022

Comments

changing urljoin to os.path.join

The behavior of urljoin is on my opionion not constructive, because of its description "Construct a full (“absolute”) URL by combining a “base URL” (base) with another URL (url). Informally, this uses components of the base URL, in particular the addressing scheme, the network location and (part of) the path, to provide missing components in the relative URL." (https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urljoin)

opened by nklever 0

Releases(v2.0.0)

v2.0.0(Jan 27, 2022)
This new major release introduces support for timezones. Also, the deprecated feature --wildcards was removed and instead the use of external tools like bzgrep on the compressed file content.log.bz2 are recommended. The content.log file was promoted to the

Changes

CONTENT_LOG_FILE is now a positional argument which defaults to STDIN when omitted

--updated-since is now a required option

New features

supply multiple CONTENT_LOG_FILEs as positional arguments to parse their combined content

--updated-since now supports timezones when specified as +HH:MM while assuming the system's time zone if not specified; date-times in content.log are always treated as UTC

Source code(tar.gz)
Source code(zip)
v1.1.0(Jan 27, 2022)
Changes

added --url-base option

deprecated --wildcards

Source code(tar.gz)
Source code(zip)
v1.0.0(Jan 27, 2022)
This is the first fully functional version. It implements minimal features with focus on readability to also act as a form of documentation around the quirks around DWD's Open Data emerging from having two seperate servers sitting behind opendata.dwd.de.

Features

List the paths contained in a content.log file for entries that have likely been updated since a given date-time

Filter those results by one or more --wildcards

Source code(tar.gz)
Source code(zip)

Owner

Deutscher Wetterdienst (DWD)

GitHub Repository

Another Provably Rare Gem Miner 💎 (for Raritygems)

Provably Rare Gem Miner Go (for Rarity) Pull Request is strongly welcome as I don't know anything about Golang/Python/Web3. Usage Install Python 3.x i

6 Apr 22, 2022

Free version of Okuru selfbot, okuru.xyz

Indigo Selfbot Free OpenSource selfbot, Premium version can be found at https://okuru.xyz (5$.) Usage python[3] main.py Installation To install you ca

31 Aug 07, 2022

Ml-design-patterns - Source code accompanying O'Reilly book: Machine Learning Design Patterns

This is not an official Google product ml-design-patterns Source code accompanying O'Reilly book: Title: Machine Learning Design Patterns Authors: Val

1.5k Jan 05, 2023

A Python feed reader library.

reader is a Python feed reader library. It aims to allow writing feed reader applications without any business code, and without enforcing a dependenc

266 Dec 30, 2022

Used the pyautogui library to automate some processes on the computer

Pyautogui Utilizei a biblioteca pyautogui para automatizar alguns processos no c

1 Dec 30, 2021

Convex Optimisation MVA course - Assignment

Convex Optimisation MVA course - Assignment This repository contains the coding files of the third assignment in the MVA Convex Optimisation course. U

1 Nov 27, 2021

Tools, guides, and resources for blockchain analysts to interface with data on the Ergo platform.

Ergo Intelligence Objective Provide a suite of easy-to-use toolkits, guides, and resources for blockchain analysts and data scientists to quickly unde

5 Mar 15, 2022

Mute your mic while you're typing. An app for Ubuntu.

Hushboard Mute your microphone while typing, for Ubuntu. Install from kryogenix.org/code/hushboard/. Installation We recommend you install Hushboard t

142 Jan 05, 2023

Python library for creating and parsing HSReplay XML files

python-hsreplay A python module for HSReplay support. https://hearthsim.info/hsreplay/ Installation The library is available on PyPI. pip install hsre

45 Mar 28, 2022

Calibre Libgen Non-fiction / Sci-tech store plugin

CalibreLibgenSci A Libgen Non-Fiction/Sci-tech store plugin for Calibre Installation Download the latest zip file release from here Open Calibre Navig

9 Dec 27, 2022

Design-by-contract in Python3 with informative violation messages and inheritance

icontract icontract provides design-by-contract to Python3 with informative violation messages and inheritance. It also gives a base for a flourishing

275 Jan 02, 2023

Fonts used to be an install-and-forget thing, but many of are now updated regularly.

Your font manager. Fonts used to be an install-and-forget thing, but many of are now updated regularly. fontman helps you keep track of the fonts you

20 Feb 07, 2022

CuraMultiplyByGrid - Cura Плагин для размножения детали сеткой на весь стол автоматически без поворота

CuraMultiplyByGrid Cura Плагин для размножения детали сеткой на весь стол автоматически без поворота. Размножение в куре настолько ужасно реализовано,

3 Dec 02, 2022

A novel dual model approach for categorization of unbalanced skin lesion image classes (Presented technical paper 📃)

1 Jan 19, 2022

Free Vocabulary Trainer - not only for German, but any language

Bilderraten DOWNLOAD THE EXE FILE HERE! What can you do with it? Vocabulary Trainer for any language Use your own vocabulary list No coding required!

4 Jan 02, 2023

Заглушки .NET библиотек для IronPython

Код репозитория основан на ironpython-stubs. Выражаю gtalarico бесконечную благодарность за вклад в развитие сообщества разработчиков скриптов и плаги

12 Nov 23, 2022

Script para generar automatización de registro de formularios IEEH

Formularios_IEEH Script para generar automatización de registro de formularios IEEH Corresponde a un conjunto de script en python que permiten la auto

1 Jan 06, 2022

KeyLogger cliente-servidor em Python para estudos

KeyLogger Esse projeto é apenas para estudos, não nos responsabilisamos por qualquer uso indevido ou prejudiciais do mesmo. Sobre O objetivo do projet

1 Dec 17, 2021

Minimalistic Gridworld Environment (MiniGrid)

Minimalistic Gridworld Environment (MiniGrid) There are other gridworld Gym environments out there, but this one is designed to be particularly simple

1.7k Jan 03, 2023

Attempt at a Windows version of the plotman Chia Plot Manager system

windows plotman: an attempt to get plotman to work on windows THIS IS A BETA. Not ready for production use just yet. Almost, but not quite there yet.

59 May 11, 2022