After fresh virtualenv install via: pipenv install git+https://github.com/bundesAPI/deutschland.git#egg=deutschland I get: Python 3.9.5 (default, Nov 23 2021, 15:27:38) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import deutschland as de from deutschland import Geo Traceback (most recent call last): File "", line 1, in ImportError: cannot import name 'Geo' from 'deutschland' (unknown location) geo = Geo() Traceback (most recent call last): File "", line 1, in NameError: name 'Geo' is not defined data = geo.fetch([52.530116236589244, 13.426532801586827], ... [52.50876180448243, 13.359631043007212]) Traceback (most recent call last): File "", line 1, in NameError: name 'geo' is not defined

Similar problem with google collab(python3.7): !pip install deutschland I get the folowing:

Attempting uninstall: urllib3 Found existing installation: urllib3 1.24.3 Uninstalling urllib3-1.24.3: Successfully uninstalled urllib3-1.24.3 Attempting uninstall: requests Found existing installation: requests 2.23.0 Uninstalling requests-2.23.0: Successfully uninstalled requests-2.23.0 Attempting uninstall: regex Found existing installation: regex 2022.6.2 Uninstalling regex-2022.6.2: Successfully uninstalled regex-2022.6.2 Attempting uninstall: numpy Found existing installation: numpy 1.21.6 Uninstalling numpy-1.21.6: Successfully uninstalled numpy-1.21.6 Attempting uninstall: Pillow Found existing installation: Pillow 7.1.2 Uninstalling Pillow-7.1.2: Successfully uninstalled Pillow-7.1.2 Attempting uninstall: pandas Found existing installation: pandas 1.3.5 Uninstalling pandas-1.3.5: Successfully uninstalled pandas-1.3.5 Attempting uninstall: lxml Found existing installation: lxml 4.2.6 Uninstalling lxml-4.2.6: Successfully uninstalled lxml-4.2.6 Attempting uninstall: beautifulsoup4 Found existing installation: beautifulsoup4 4.6.3 Uninstalling beautifulsoup4-4.6.3: Successfully uninstalled beautifulsoup4-4.6.3 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. xarray-einstats 0.2.2 requires numpy>=1.21, but you have numpy 1.19.0 which is incompatible. tensorflow 2.8.2+zzzcolab20220527125636 requires numpy>=1.20, but you have numpy 1.19.0 which is incompatible. google-colab 1.0.0 requires requests~=2.23.0, but you have requests 2.28.1 which is incompatible. datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible. albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible. Successfully installed Pillow-8.4.0 beautifulsoup4-4.11.1 boto3-1.24.34 botocore-1.27.34 dateparser-1.1.1 de-autobahn-1.0.4 de-bundesrat-0.1.0 de-bundestag-0.1.0 de-dwd-1.0.1 de-interpol-0.1.0 de-jobsuche-0.1.0 de-ladestationen-1.0.5 de-mudab-0.1.0 de-nina-1.0.2 de-polizei-brandenburg-0.1.0 de-risikogebiete-0.1.0 de-smard-0.1.0 de-strahlenschutz-1.0.0 de-travelwarning-0.1.0 de-zoll-0.1.0 deutschland-0.3.0 gql-2.0.0 graphql-core-2.3.2 jmespath-1.0.1 lxml-4.9.1 mapbox-vector-tile-1.2.1 numpy-1.19.0 onnxruntime-1.10.0 pandas-1.1.5 pyclipper-1.3.0.post3 pypresseportal-0.1 regex-2022.3.2 requests-2.28.1 rx-1.6.1 s3transfer-0.6.0 slugify-0.0.1 urllib3-1.26.10

WARNING: The following packages were previously imported in this runtime: [PIL,numpy] You must restart the runtime in order to use newly installed versions.

import deutschland as de geo = de.Geo() AttributeError Traceback (most recent call last)

in () 1 import deutschland as de ----> 2 geo = de.Geo() AttributeError: module 'deutschland' has no attribute 'Geo'

Running an older version: deutschland = "==0.1.9" I don't get the import errors in Geo() but an empy result:

Python 3.9.13 (main, May 23 2022, 22:01:06) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information.

from deutschland import Geo geo = Geo() data = geo.fetch([52.530116236589244, 13.426532801586827], ... [52.50876180448243, 13.359631043007212]) print(data.keys()) dict_keys([])

With the Bundesanzeiger I get the import error again:

from deutschland import Bundesanzeiger ba = Bundesanzeiger() 2022-07-22 00:09:52.683533: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory 2022-07-22 00:09:52.683555: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. Traceback (most recent call last): File "", line 1, in File "/home/moritz/.local/share/virtualenvs/deutschland-fqErnsp1/lib/python3.9/site-packages/deutschland/bundesanzeiger/bundesanzeiger.py", line 47, in init self.model = deutschland.bundesanzeiger.model.load_model() File "/home/moritz/.local/share/virtualenvs/deutschland-fqErnsp1/lib/python3.9/site-packages/deutschland/bundesanzeiger/model.py", line 36, in load_model return keras.models.load_model( File "/home/moritz/.local/share/virtualenvs/deutschland-fqErnsp1/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler raise e.with_traceback(filtered_tb) from None File "/home/moritz/.local/share/virtualenvs/deutschland-fqErnsp1/lib/python3.9/site-packages/keras/saving/save.py", line 206, in load_model raise IOError(f'No file or directory found at {filepath_str}') OSError: No file or directory found at assets/model.h5 data = ba.get_reports("Deutsche Bahn AG") Traceback (most recent call last): File "", line 1, in NameError: name 'ba' is not defined

Could you point to what I am doing wrong? Best regards

Anbei noch ein kleiner PR um die Details einer Bekanntmachung zu holen. Ich bitte um Feedback.

Als nächstes steht jetzt die hole Firma mit Bekanntmachungen/Veröffentlichungen auf der Liste.

Draft for merging all optional packages into the Deutschland package.

This probably needs a bit of testing before it can be released.

• Should we make all subpackages optional? This is how it is now with deutschland[all] all packages will get installed
• This might be a breaking change, since an update will probably remove packages which were part of the package before (not tested yet)
• How is the version resolving? Does it take super long everywhere? Or only with poetry and on my machine?
• Can we make the package with tensorflow as dependency also optional and independent of the deutschland package? Thats a huge dependency.

@LilithWittmann let me know what you think :+1:

I was just trying to install the this package and couldn't get it to work using python 3.9 on Windows 10. Mostly due to numpy/blas/lapack/mkl errors somewhere deep down. However, reading the pyproject.toml I saw the mentioning of python 3.6.2. => Using python 3.6 I was able to install this package without errors.

Some functions didn't work because they required a newer version of numpy.

The newer version of numpy also requires python to be at least version 3.8 - that's why I also upgraded that requirement.

I ran the sample code, but had the following error:

OSError Traceback (most recent call last) in () 1 from deutschland import Bundesanzeiger ----> 2 ba = Bundesanzeiger() 3 # search term 4 data = ba.get_reports("Deutsche Bahn AG") 5 # returns a dictionary with all reports found as fulltext reports

3 frames /usr/local/lib/python3.7/dist-packages/keras/saving/save.py in load_model(filepath, custom_objects, compile, options) 202 if isinstance(filepath_str, str): 203 if not tf.io.gfile.exists(filepath_str): --> 204 raise IOError(f'No file or directory found at {filepath_str}') 205 206 if tf.io.gfile.isdir(filepath_str):

OSError: No file or directory found at assets/model.h5

Hi,

I have exported your model to solve the CAPTCHAs to the ONNX format. This has the advantage that you get rid of TensorFlow as a dependency (~500 MB) and can use onnxruntime (~5 MB) for inference instead – would make the whole project way more lightweight. There are also significantly fewer problems updating onnxruntime than TensorFlow without breaking the model.

And I could also fix https://github.com/bundesAPI/deutschland/issues/8, if you're interested.

In a map oriented to the north, the first coordinate passed to Geo.fetch() is the lower left, and the second coordinate is the upper right of the two coordinates.

The following image shows both coordinates from the documentation in OpenStreetMap.

# top_right and bottom_left coordinates
data = geo.fetch([52.50876180448243, 13.359631043007212], 
                 [52.530116236589244, 13.426532801586827])

Maybe I misunderstand the names top_right and bottom_left. Please correct me if this is the case. Otherwise, I would work on a pull request for this issue, which renames the variables.

Man man, wat'n Krampf das Ganze. Aber gut. Hier ist er, der Such-Gerät.

Schön ist das Ganze zwar nicht, aber hier ist ein Vorschlag für die erste Funktionalität des Handelsregisters:

Die Suche nach Firmen im Handelsregister

Ich habe davon abgesehen jeden Suchparameter aus dem Deutschen ins Englische zu übersetzen und biete stattdessen die Möglichkeit an, jeden Parameter in einem Dict nach Belieben zu setzen und damit den Default zu überschreiben. Ich habe versucht die Parameter so gut es geht zu dokumentieren. Außerdem kann man die Bundesländer als gesonderten Parameter als Liste angeben.

Ein große Einschränkung hat das Ganze allerdings noch: Pagination wird noch nicht unterstützt. Sprich, bisher werden nur die ersten 100 Ergebnisse zurückgegeben. Ich wollte erstmal diesen PR anbieten, bevor ich mich mit der Pagination weiter beschäftige.

Ich bitte einmal um Feedback. Bitte auch über die Entscheidung, ob diese Funktionalität bereits gemerged werden kann, oder ob wir später alle Funktionalitäten (sprich auch Bekanntmachungen und Veröffentlichungen) zusammen mergen bzw. releasen möchten.

Ebenfalls habe ich die Docs vorerst in reStructuredText-Format geschrieben. Bitte auch hier einmal entscheiden, welches Format gewünscht ist.

I rewrote bundesanzeiger to use requests instead of selenium, this should boost performance and portability.

Note: I could not get the normal branch to work properly, so I did not fully cross-check the output.

I am receiving following error message, it looks like the model used to solve the captchas does not exist in assets/model.h5 directory: OSError: SavedModel file does not exist at: assets/model.h5/{saved_model.pbtxt|saved_model.pb}

Here's my code:

from deutschland import Bundesanzeiger
ba = Bundesanzeiger()
# search term
data = ba.get_reports("Deutsche Bahn AG")
# returns a dictionary with all reports found as fulltext reports
print(data.keys())
# dict_keys(['Jahresabschluss zum Geschäftsjahr vom 01.01.2020 bis zum 31.12.2020', 'Konzernabschluss zum Geschäftsjahr vom 01.01.2020 bis zum 31.12.2020\nErgänzung der Veröffentlichung vom 04.06.2021',

Some reports share the same names, e.g. Mitteilung von Netto-Leerverkaufspositionen. The problem is that get_reports cannot return multiple items with the same name because it returns a dictionary/map.

#!/opt/homebrew/bin/python3
from deutschland.bundesanzeiger import Bundesanzeiger
ba = Bundesanzeiger()
data = ba.get_reports("DE000A0TGJ55")
print(data.keys())

print(data["Mitteilung von Netto-Leerverkaufspositionen"])

dict_keys(['Mitteilung von Netto-Leerverkaufspositionen'])
{'date': datetime.datetime(2022, 9, 26, 0, 0), 'name': 'Mitteilung von Netto-Leerverkaufspositionen', 'company': 'BlackRock Investment Management (UK) Limited', 'report': '\n\n\n\n\xa0\n\n\n\n\n\n\n\nBlackRock Investment Management (UK) Limited\nLondon\nMitteilung von Netto-Leerverkaufspositionen\nZu folgendem Emittenten wird vom oben genannten Positionsinhaber eine Netto-Leerverkaufsposition\n            gehalten:\n\nVARTA AKTIENGESELLSCHAFT\n\n\nISIN: DE000A0TGJ55\n\nDatum der Position: 23.09.2022\nProzentsatz des ausgegebenen Aktienkapitals: 2,26 %\n\xa0\n\n\n\n\n\n\n\n\n\n\n\n\n'}

Here you can see how it looks on the website:

The tests fail under windows as can be seen here

The reason is under windows somehow the telephon symbol is not parsed properly.

When I tried to print it I saw something like

E       UnicodeEncodeError: 'charmap' codec can't encode character '\u260e' in position 38: character maps to <undefined>

Unfortunately I do not have access to a windows machine to dig deeper in a reasonable manner.

Raw error:

================================== FAILURES ===================================
_________________________________ test_verena _________________________________

    def test_verena():
        v = Verena()
>       res = v.get()

tests\verena\test_verena.py:6: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\deutschland\verena\verena.py:21: in get
    extract = VerenaExtractor(page).extract()
C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\deutschland\verena\verenaextractor.py:38: in extract
    phone, fax, homepage, email, deadline = self.__extract_part4(aus_parts[3])
C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\deutschland\verena\verenaextractor.py:158: in __extract_part4
    print(x)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <encodings.cp1252.IncrementalEncoder object at 0x000002AC31A54F10>
input = '\\r\\r\\n\\r\\r\\n                                \u260e 02381 973060\\r\\r\\n                                '
final = False

    def encode(self, input, final=False):
>       return codecs.charmap_encode(input,self.errors,encoding_table)[0]
E       UnicodeEncodeError: 'charmap' codec can't encode character '\u260e' in position 38: character maps to <undefined>

C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\encodings\cp1252.py:19: UnicodeEncodeError
___________________________ test_extractor_content ____________________________

    def test_extractor_content():
        with open("tests/verena/ausschreibung_test_input.html", "r") as f:
            with open("tests/verena/ausschreibung_correct_result.json", "r") as correct:
                content = "<html><body>" + f.read() + "</body></html>"
                ve = VerenaExtractor(content)
                res = ve.extract()
>               assert len(res) == 1 and res[0] == json.loads(correct.read())
E               AssertionError: assert (1 == 1 and {'comments': ...ulingen', ...} == {'comments': ...ulingen', ...}
E                +  where 1 = len([{'comments': 'Bemerkung zur Stelle: Testbemerkung', 'contact': {'fax': '0172 2222 2222', 'homepage': 'http://www.eine...line/': '17.09.2021', 'desc': 'Eine Schule\nSchule der Sekundarstufe II\ndes Landkreis Schuling\n9999 Schulingen', ...}])
E                 Omitting 11 identical items, use -vv to show
E                 Differing items:
E                 {'contact': {'fax': '0172 2222 2222', 'homepage': 'http://www.eine-schule.de/', 'mail': {'adress': '[email protected]...'mailto:[email protected]?subject=Stellenausschreibung in VERENA', 'subject': 'Stellenausschreibung in VERENA'}}} != {'contact': {'fax': '0172 2222 2222', 'homepage': 'http://www.eine-schule.de/', 'mail': {'adress': '[email protected]?subject=Stellenausschreibung in VERENA', 'subject': 'Stellenausschreibung in VERENA'}, 'phone': '0172 1111 1111'}}
E                 Full diff:
E                   {
E                    'comments': 'Bemerkung zur Stelle: Testbemerkung',
E                    'contact': {'fax': '0172 2222 2222',
E                                'homepage': 'http://www.eine-schule.de/',
E                                'mail': {'adress': '[email protected]',
E                                         'raw': 'mailto:[email protected]?subject=Stellenausschreibung '
E                                                'in VERENA',
E                 -                       'subject': 'Stellenausschreibung in VERENA'},
E                 +                       'subject': 'Stellenausschreibung in VERENA'}},
E                 ?                                                                   +
E                 -              'phone': '0172 1111 1111'},
E                    'deadline': '17.09.2021',
E                    'desc': 'Eine Schule\n'
E                            'Schule der Sekundarstufe II\n'
E                            'des Landkreis Schuling\n'
E                            '9999 Schulingen',
E                    'duration': '01.01.2021 - 01.01.2022',
E                    'geolocation': {'coord_system': 'epsg:25832',
E                                    'coordinates': [1111111,
E                                                    1111111],
E                                    'post_adress': 'Eine Straße 1\n'
E                                                   '99999 Schulingen'},
E                    'hours_per_week': '13,5',
E                    'replacement_job_title': 'Lehrkraft',
E                    'replacement_job_type': 'Vertretung',
E                    'replacement_job_type_raw': 'Vertretung für',
E                    'school_id': '99999',
E                    'subjects': ['Fach 1',
E                                 'Fach 2'],
E                   })

tests\verena\test_verenaextractor.py:12: AssertionError
============================== warnings summary ===============================

Hello, when trying to use the destatis api (timeseries_data to be exact), it throws a SSLCertVerificationError.

MaxRetryError: HTTPSConnectionPool(host='www-genesis.destatis.de', port=443): Max retries exceeded with url: /genesisWS/rest/2020/data/timeseries?username=*******&password=******* (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')))

My code is just the example for timeseries_data, with a valid username and password provided. The host url is https://www-genesis.destatis.de/genesisWS/rest/2020.

In order to reach the widest audience, I think we should be able to make a statement about which python versions on which OS we can support.

For this I suggest we also run the tests on windows and mac.

Here are some stats which os are used to download the package.

Hi, I would like to extract all the charging stations in Germany. I guess the ladestationen would give me this info. However, I'm not sure what "geometry" should be provided as input here. Also, the URI is localhost. Could you please provide a valid URI?

Thanks