In GitLab by @rob.speer on May 16, 2018, 01:23

Suppose I have a Web application, myapp, that needs to serve a static file, which by convention needs to be in the path myapp/static/ld/context.ld.json. Suppose I also want to be able to access that file from Python code, because its contents are used in a test.

As importlib_resources is currently defined, I would need to rewrite the path as if it were a Python submodule, even though it does not contain actual Python code: path(myapp.static.ld, "context.ld.json"). I would also need to create empty files named myapp/static/__init__.py and myapp/static/ld/__init__.py, and hopefully exclude them from being served as static files.

That would be enough for me to give up and use paths relative to __file__ instead. In general, I would heartily recommend importlib if I could reasonably promise that it was an improvement over using __file__ or over existing uses of pkg_resources, which it wouldn't be if it doesn't support subdirectories.

The call I would like to be able to make in this situation is path(myapp, "static/ld/context.ld.json").

This kind of release process is a huge nuisance for those of us packaging software downstream. Not being able to use PyPI tarballs or even GitHub tarballs is requires us to go well out of our way to accomodate your package. Please just conform to normal Python packaging norms.

In GitLab by @jaraco on Nov 2, 2018, 02:48

Attempting to retrieve resources from a namespace package fails.

draft $ mkdir foo
draft $ touch foo/bar.txt
draft $ rwt importlib_resources
Collecting importlib_resources
  Using cached https://files.pythonhosted.org/packages/2f/f7/b4aa02cdd3ee7ebba375969d77c00826aa15c5db84247d23c89522dccbfa/importlib_resources-1.0.2-py2.py3-none-any.whl
Installing collected packages: importlib-resources
Successfully installed importlib-resources-1.0.2
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib_resources
>>> importlib_resources.read_text(__import__('foo'), 'bar.txt')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/resources.py", line 169, in read_text
    with open_text(package, resource, encoding, errors) as fp:
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/resources.py", line 126, in open_text
    _check_location(package)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/resources.py", line 82, in _check_location
    raise FileNotFoundError(f'Package has no location {package!r}')
FileNotFoundError: Package has no location <module 'foo' (namespace)>

I see an obvious problem here - that a namespace package can have more than one base path, so it has no single location. But it does have a location... and pkg_resources lets one load resources from namespace package:

draft $ cat > foo/__init__.py
import pkg_resources; pkg_resources.declare_namespace('foo')
draft $ python
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>> pkg_resources.resource_stream('foo', 'bar.txt')
<_io.BufferedReader name='/Users/jaraco/draft/foo/bar.txt'>

pkg_resources doesn't succeed with a PEP 420 namespace package:

$ rm foo/__init__.py
>>> pkg_resources.resource_stream('foo', 'bar.txt')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1145, in resource_stream
    return get_provider(package_or_requirement).get_resource_stream(
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 359, in get_provider
    return _find_adapter(_provider_factories, loader)(module)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1387, in __init__
    self.module_path = os.path.dirname(getattr(module, '__file__', ''))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/posixpath.py", line 156, in dirname
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType

But even in that situation, it does allow for loading resources for a module within a PEP 420 namespace package:

draft $ touch foo/mod.py
draft $ tree
.
└── foo
    ├── bar.txt
    └── mod.py
draft $ python
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>> pkg_resources.resource_stream('foo.mod', 'bar.txt')
<_io.BufferedReader name='/Users/jaraco/draft/foo/bar.txt'>

This issue sort-of relates to #60, but is more serious because it seems there's no input that importlib_resources can accept to load resources from a namespace package.

The issue emerged in pmxbot when I tried to convert the package from a (deprecated) pkg_resources-style namespace package to a PEP 420 namespace package. The code failed at this line when it tried to load the phrases.

It seems to me (without looking at the code) it should be straightforward to support loading resources from namespace packages, following the same logic that importlib follows to import a module from that package.

The only other option I see is to force packages to rewrite their packages to only put resources in non-namespace packages, which is a bit of an imposition and unintuitive constraint.

In GitLab by @warsaw on Nov 13, 2017, 12:27

Over the weekend, I wrote an experimental branch to switch Mailman 3 to importlib_resources. Sorry for the catch-all issue, but I want to at least capture my thoughts on the current API. We can turn these into separate issues if/when we want to change anything. And at least this should serve as a good conversion guide for #16 .

• open() isn't a very good name, especially if you tend to use from importlib_resources import style imports. I do like the shorter name, but the problem is obvious: this shadows the built-in open() function. In my branch, I've used from importlib_resources import open as resources_open but that's less than ideal. As open() is also the obvious choice for the function name, I don't yet have a good suggested alternative.

• path() returning a pathlib.Path is problematic. It's likely the right thing to do, but it means that conversion requires the liberal use of str() wrapping, since much existing code is expecting a concrete str object. Maybe this is better in 3.7 (I only tested with 3.6) for stdlib APIs, but won't help you much for internal project APIs that expect a string. Interestingly, this tends to be a bit easier if you use an ExitStack rather than a with-statement. Compare:

        with path('mailman.testing', 'config-with-instances.pck') as pckpath:
            pckfile = str(pckpath)

and

        with ExitStack() as resources:
            pckfile = str(resources.enter_context(
                path('mailman.testing', 'config.pck')))

• Returning a context from path() is often inconvenient. Of course, I understand why we have to do that, but ensuring that the context is properly cleaned up can be tricky, and it's certainly much less convenient at times. I've had to change a few internal APIs to accept an ExitStack argument, and then add the path context to that, since the resources have to be managed several call frames higher. E.g.

@public
def use_test_organizational_data(resources):
    # Point the organizational URL to our test data.
    filename = str(resources.enter_context(
        path('mailman.rules.tests.data', 'org_domain.txt')))
    url = 'file:///{}'.format(filename)
    return resources.enter_context(
        configuration('dmarc', org_domain_data_url=url))

• Missing API. I couldn't totally rid myself of pkg_resources because there's no current alternative to resource_isdir() and resource_listdir().

• read() should support binary reads. I know we have open() for that, but it turns out to be convenient if read(encoding=None) returned bytes. I know we talked about that before, but it's become more obvious when using this in real-world code. Conversely, it's still a bit inconvenient that open() only returns a file opened in binary mode. Fortunately, there's email.message_from_binary_file() where I was previously using email.message_from_file(), so it wasn't difficult to fix, but it does mean that converting from pkg_resources to importlib_resources is less mechanical.

In GitLab by @goshansp on Aug 21, 2020, 09:38

Scope

We are unable to upload or sync importlib_resources>=1.1.0 into our python repository (rhel7-pulp2.21-python2.7.5). Until version 1.0.2 this went fine.

Workaround rhel7-pulp2.21-python2.7.5 (Manual Upload to Pulp)

1. delete / create pytho repo and dont add importlib_resources via feed
2. download importlib_resources-1.0.2 from pypi
3. upload / publish
4. profit

other packages that dont cause said issue

ansible,ansible-base,ansible-lint,ansible-tower-cli,anyconfig,arrow,asn1crypto,aspy-yaml,atomicwrites,attrs,babel,backports-ssl-match-hostname,bcrypt,binaryornot,cerberus,certifi,cffi,cfgv,chardet,click,click_help_colors,click-completion,commonmark,colorama,configparser,contextlib2,cookiecutter,cryptography,dataclasses,debtcollector,distro,dnspython,docker,docker-py,docker-pycreds,dparse,entrypoints,enum34,fasteners,flake8,funcsigs,functools32,future,git-url-parse,identify,idna,importlib-metadata,infoblox-client,ipaddress,iso8601,jinja2,jinja2-time,jmespath,lxml,markupsafe,mccabe,molecule,monotonic,more-itertools,msgpack,netaddr,netifaces,nodeenv,ntlm-auth,oslo-config,oslo-context,oslo-i18n,oslo-log,oslo-serialization,oslo-utils,packaging,paramiko,pathlib2,pathspec,pbr,pexpect,pip,pluggy,poyo,pre-commit,prometheus-client,psutil,ptyprocess,py,pyasn1,pycodestyle,pycparser,pydf,pygments,pyflakes,pyinotify,pykerberos,pynacl,pyopenssl,pyparsing,pytest,python-dateutil,python-gilt,python-gitlab,python-memcached,python-slugify,pytz,pyup-tools,pyvmomi,pywinrm,pyyaml,requests,requests-kerberos,requests-ntlm,rich,rfc3986,ruamel-ordereddict,ruamel-yaml,ruamel-yaml-clib,safety,selinux,scandir,setuptools,sh,shellingham,six,stevedore,tabulate,testinfra,text-unidecode,toml,tree-format,typing,typing-extensions,urllib3,virtualenv,wcwidth,websocket-client,whichcraft,wrapt,xmltodict,yamllint,yq,zipp

suspected cause

There seems to be incompatibilities with the versioning. Possibly the importer gets 0.0.0 from two packages resulting in an error during sync. This may be due to setuptools==0.9.8

reproduction pulp sync fail

 $ pulp-admin python repo create $pulp_proxy --repo-id python-test --feed https://pypi.org/ --package-names importlib_resources
 $ pulp-admin python repo sync run --repo-id python-test
+----------------------------------------------------------------------+
                 Synchronizing Repository [python-test]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading Python metadata.
[-]
... completed

Copying units already in pulp
[\]
... completed

Downloading and processing Python packages.
[\]
... completed

Task Succeeded

Task Failed

[Errno 17] File exists:
'/var/cache/pulp/[email protected]/c895ee09-f18d-417
8-85d6-95d26b0a5bc8/python-test/simple/importlib-resources'
$ rpm -qa | grep pulp
pulp-python-plugins-2.0.3-1.el7.noarch
pulp-server-2.21.0-1.el7.noarch
pulp-puppet-plugins-2.21.0-1.el7.noarch
python-pulp-bindings-2.21.0-1.el7.noarch
pulp-consumer-client-2.21.0-1.el7.noarch
pulp-puppet-admin-extensions-2.21.0-1.el7.noarch
pulp-selinux-2.21.0-1.el7.noarch
python-pulp-python-common-2.0.3-1.el7.noarch
python-pulp-rpm-common-2.21.0-1.el7.noarch
python-pulp-agent-lib-2.21.0-1.el7.noarch
python-pulp-oid_validation-2.21.0-1.el7.noarch
pulp-rpm-plugins-2.21.0-1.el7.noarch
pulp-rpm-admin-extensions-2.21.0-1.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
pulp-docker-admin-extensions-3.2.5-1.el7.noarch
pulp-python-admin-extensions-2.0.3-1.el7.noarch
python-pulp-repoauth-2.21.0-1.el7.noarch
python-pulp-client-lib-2.21.0-1.el7.noarch
pulp-agent-2.21.0-1.el7.noarch
pulp-rpm-handlers-2.21.0-1.el7.noarch
pulp-docker-plugins-3.2.5-1.el7.noarch
python-pulp-common-2.21.0-1.el7.noarch
pulp-admin-client-2.21.0-1.el7.noarch
pulp-rpm-consumer-extensions-2.21.0-1.el7.noarch
python-pulp-docker-common-3.2.5-1.el7.noarch
python-pulp-puppet-common-2.21.0-1.el7.noarch
pulp-rpm-yumplugins-2.21.0-1.el7.noarch

journal -f / publish fail

Aug 26 10:34:49 redacted pulp[1866]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[2e2d5a08-05e3-4673-9fcc-4bce3b8b86da]
Aug 26 10:34:49 redacted pulp[2223]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[8266d2ef-391c-41a1-8b6d-c4ef8454d8a1]
Aug 26 10:34:49 redacted pulp[3098]: celery.app.trace:INFO: [2e2d5a08] Task pulp.server.async.tasks._queue_reserved_task[2e2d5a08-05e3-4673-9fcc-4bce3b8b86da] succeeded in 0.0141177220503s: None
Aug 26 10:34:49 redacted pulp[2223]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[28b04604-c48c-4d50-91d9-083c3429fb57]
Aug 26 10:34:49 redacted pulp[3074]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): pypi.org
Aug 26 10:34:49 redacted pulp[3074]: nectar.downloaders.threaded:INFO: Download succeeded: https://pypi.org/pypi/importlib_resources/json.
Aug 26 10:34:49 redacted pulp[3074]: pulp_python.plugins.importers.sync:INFO: Processing metadata retrieved from https://pypi.org/pypi/importlib_resources/json.
Aug 26 10:34:51 redacted pulp[1866]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[f2abd34d-6883-498d-9d39-49a7b976f3d4]
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:INFO: [8266d2ef] Task pulp.server.managers.repo.sync.sync[8266d2ef-391c-41a1-8b6d-c4ef8454d8a1] succeeded in 2.19935470994s: <pulp.server.async.tasks.TaskResult object at 0x7f96e4aebc10>
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:INFO: [28b04604] Task pulp.server.async.tasks._release_resource[28b04604-c48c-4d50-91d9-083c3429fb57] succeeded in 0.00107404403389s: None
Aug 26 10:34:51 redacted pulp[2223]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[c895ee09-f18d-4178-85d6-95d26b0a5bc8]
Aug 26 10:34:51 redacted pulp[3098]: celery.app.trace:INFO: [f2abd34d] Task pulp.server.async.tasks._queue_reserved_task[f2abd34d-6883-498d-9d39-49a7b976f3d4] succeeded in 0.015954433009s: None
Aug 26 10:34:51 redacted pulp[2223]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[a8098104-c1e6-45a4-bd0b-5850aef21e40]
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488) Exception caught from plugin during publish for repo [python-test]
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488) Traceback (most recent call last):
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1259, in _do_publish
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 894, in wrap_f
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     return f(*args, **kwargs)
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/web.py", line 94, in publish_repo
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     return self._publisher.process_lifecycle()
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     super(PluginStep, self).process_lifecycle()
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     step.process()
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 257, in process
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     self._process_block()
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 304, in _process_block
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     self.process_main()
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/steps.py", line 69, in process_main
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     self.write_simple_api(projects)
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/steps.py", line 100, in write_simple_api
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     PublishMetadataStep._create_project_index(project_name, simple_path, packages)
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/steps.py", line 118, in _create_project_index
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     os.makedirs(project_path)
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib64/python2.7/os.py", line 157, in makedirs
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     mkdir(name, mode)
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488) OSError: [Errno 17] File exists: '/var/cache/pulp/[email protected]/c895ee09-f18d-4178-85d6-95d26b0a5bc8/python-test/simple/importlib-resources'
Aug 26 10:34:51 redacted pulp[3074]: pulp.server.async.tasks:INFO: [c895ee09] Task failed : [c895ee09-f18d-4178-85d6-95d26b0a5bc8]
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488) Task pulp.server.managers.repo.publish.publish[c895ee09-f18d-4178-85d6-95d26b0a5bc8] raised unexpected: OSError(17, 'File exists')
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488) Traceback (most recent call last):
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     R = retval = fun(*args, **kwargs)
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 686, in __call__
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     return super(Task, self).__call__(*args, **kwargs)
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 108, in __call__
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     return super(PulpTask, self).__call__(*args, **kwargs)
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     return self.run(*args, **kwargs)
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1110, in publish
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1207, in check_publish
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1259, in _do_publish
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 894, in wrap_f
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     return f(*args, **kwargs)
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/web.py", line 94, in publish_repo
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     return self._publisher.process_lifecycle()
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     super(PluginStep, self).process_lifecycle()
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     step.process()
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 257, in process
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     self._process_block()
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 304, in _process_block
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     self.process_main()
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/steps.py", line 69, in process_main
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     self.write_simple_api(projects)
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/steps.py", line 100, in write_simple_api
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     PublishMetadataStep._create_project_index(project_name, simple_path, packages)
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/steps.py", line 118, in _create_project_index
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     os.makedirs(project_path)
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib64/python2.7/os.py", line 157, in makedirs
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     mkdir(name, mode)
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488) OSError: [Errno 17] File exists: '/var/cache/pulp/[email protected]/c895ee09-f18d-4178-85d6-95d26b0a5bc8/python-test/simple/importlib-resources'
Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:INFO: [a8098104] Task pulp.server.async.tasks._release_resource[a8098104-c1e6-45a4-bd0b-5850aef21e40] succeeded in 0.00126324000303s: None

reproduction steps pulp upload fail

The following will fail for any version >=1.1.0

$ pulp-admin -vvvvv python repo upload --repo-id python-test --file importlib_resources-1.1.0.tar.gz
...
Task Failed

The importer python_importer indicated a failed response when uploading
python_package unit to repository python-test. Summary: 'expected string or
buffer'. Details: '{}'

In GitLab by @aleskva on Mar 1, 2020, 10:23

See the following log from our Windows tests: https://ci.appveyor.com/project/ladsgroup/pywikibot-g4xqx/build/job/w3ygc2f3winb4nnb

$ pip install virtualenv
Collecting virtualenv
  Downloading https://files.pythonhosted.org/packages/c1/61/7506ddd79ef6f09beeefb81c4c55bf395a8ad96b33ff1c6b06e40f8aa101/virtualenv-20.0.7-py2.py3-none-any.whl (8.0MB)
Collecting filelock<4,>=3.0.0 (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl
Collecting distlib<1,>=0.3.0 (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/7d/29/694a3a4d7c0e1aef76092e9167fbe372e0f7da055f5dcf4e1313ec21d96a/distlib-0.3.0.zip (571kB)
Collecting six<2,>=1.9.0 (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
Collecting importlib-metadata<2,>=0.12 (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/8b/03/a00d504808808912751e64ccf414be53c29cad620e3de2421135fcae3025/importlib_metadata-1.5.0-py2.py3-none-any.whl
Collecting appdirs<2,>=1.4.3 (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/56/eb/810e700ed1349edde4cbdc1b2a21e28cdf115f9faf263f6bbf8447c1abf3/appdirs-1.4.3-py2.py3-none-any.whl
Collecting importlib-resources<2,>=1.0 (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/a4/23/ffca4a0003519a156b178b3edd98ac9c9de051af76a993d15ed19862ea4d/importlib_resources-1.1.0-py2.py3-none-any.whl
Collecting zipp>=0.5 (from importlib-metadata<2,>=0.12->virtualenv)
  Downloading https://files.pythonhosted.org/packages/6f/6d/a55f6e81ac213942b9a19cbc05b560c726c3e16f8fb17555f059c17d65f2/zipp-3.0.0-py3-none-any.whl
Installing collected packages: filelock, distlib, six, zipp, importlib-metadata, appdirs, importlib-resources, virtualenv
  Running setup.py install for distlib
Successfully installed appdirs-1.4.3 distlib-0.3.0 filelock-3.0.12 importlib-metadata-1.5.0 importlib-resources-1.1.0 six-1.14.0 virtualenv-20.0.7 zipp-3.0.0
You are using pip version 7.1.2, however version 20.0.2 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
$ virtualenv env
ImportError: cannot import name 'ContextManager'

In GitLab by @jaraco on Mar 24, 2020, 11:36

As discussed in #90, this MR attempts to implemented the proposed interface for loaders to supply more basic functionality for loading resources.

In GitLab by @jaraco on Sep 11, 2018, 21:09

Merges feature/mechanized-releases -> master

Based on the work in python-devs/importlib_metadata#8, follow the same pattern here.

I'm marking this a WIP while I work out issues and consider python-devs/importlib_metadata!9.

In GitLab by @jokerjokerer on Jan 14, 2020, 12:48

Within a zipapp the loaders archive path will be exactly the path passed on the caller side. So for:

py -3.5 tasks/virtualenv.pyz

the zipimporter archive path will be tasks/virtualenv.py.

https://gitlab.com/python-devs/importlib_resources/blob/1.0.2/importlib_resources/_py3.py#L83 sets the resource path always absolute. The logic for the zipimporter seems to care about this discrepancy (https://github.com/python/cpython/blob/v3.5.9/Modules/zipimport.c#L1131) as only paths that are starting with loader.archive will be returned.

For example for resource tasks/virtualenv.py/a.data:

• fails absolute format /w/tasks/virtualenv.py/a.data
• succeeds tasks/virtualenv.py/a.data
• succeeds a.data

The current code returns the first format though. Shouldn't we normalize paths relative to the archive at https://gitlab.com/python-devs/importlib_resources/blob/1.0.2/importlib_resources/_py3.py#L93?

In GitLab by @warsaw on Dec 6, 2017, 11:40

Merges pep518 -> master

Make the pyproject.toml file the single source of truth about the project metadata. This should let us flit publish the package when the time comes.

The trick is that tox still requires the setup.py, but I have added a new update-setup.py script that generates that from the pyproject.toml. Yes, it does evil things but it works. Based on a suggestion by upstream flit.

I have a regression between version 5.7.1 and 5.8.0. I think the cause is this commit.

I have a reproduction scenario in the zip. To reproduce:

• unzip repro.zip
• cd repro
• pip install -e .
• python -m test.test --info

If you change the setup.cfg to use version 5.7.1, no error is raised repro.zip

Hi,

I would like to import a file which is in the root of my program.

I read the docs but didn't find an answer : https://importlib-resources.readthedocs.io/en/latest/using.html I would like to know how to import the "resource3.txt" file of the doc's example.

Someone else on Stack Overflow has a similar problem. https://stackoverflow.com/q/73559659/9665509

Could you help me ? I apologize as it's not really an issue.

• When MultiplexedPath.joinpath is passed a subdirectory (tree) that is common to multiple of the parent's paths, joinpath will return a MultiplexedPath. Fixes #265

I ran into the following issue while trying to update my MultiplexedPath example in #264 to the current structure. Let's say you have the following path structure:

data01
  |_x

data02
  |_x
    |_y
      |_z.txt

If you combine data01 and data02 in aMultiplexedPath, data01/x will make it impossible to find data02/x/y, see the following example:

from importlib_resources.readers import MultiplexedPath

p = MultiplexedPath('data01', 'data02')

In this case if you do:

print(p / 'x/y')

this will give data01/x/y instead of data02/x/y, and

for f in (p / 'x/y').iterdir():
    print(f)

will raise a FileNotFoundError.

The reason for this is that in Traversable.joinpath(), only the first path segment is used to search for a match:

names = itertools.chain.from_iterable(
    path.parts for path in map(pathlib.PurePosixPath, descendants)
)
target = next(names)

and then, only the first match is returned:

matches = (
    traversable for traversable in self.iterdir() if traversable.name == target
)
try:
    match = next(matches)

The solution to this problem could be related to the suggestion in #264, unless returning MultiplexedPaths isn't the desired behaviour of MultiplexedPath.joinpath().

Do the importlib.resources documentation (or maybe packaging guides) discuss any of these use cases and best practices?

I just reviewed the docs, and they are pretty sparse about the purpose of the API. It does link to the "using" guide, and that does seem like it might be an appropriate place to publish that information.

Originally posted by @jaraco in https://github.com/python/importlib_resources/issues/68#issuecomment-1199476054

This happens to me when I try to access resources in a native namespace package.

Tested on:

• Windows, Python 3.10.4
• Docker in WSL2, image python:3.10-slim, Python 3.10.5

The reason seems to be that importlib_resources.files() returns MultiplexedPath from stdlib (importlib.readers) rather than from importlib_resources.readers.

To recreate:

1. Install any namespace package, e.g. sphinxcontrib-htmlhelp;

   pip install sphinxcontrib-htmlhelp
   

2. Use files with joinpath to get the path of a resource.

   from importlib_resources import files
   
   print(root:=files('sphinxcontrib'), 'from', root.__module__)
   print(root.joinpath('htmlhelp', 'templates'))
   

Expected output (which I get with Pythons 3.7-3.9):

MultiplexedPath('/usr/local/lib/python3.9/site-packages/sphinxcontrib') from importlib_resources.readers
/usr/local/lib/python3.9/site-packages/sphinxcontrib/htmlhelp/templates

Bug with Pythons 3.10:

MultiplexedPath('/usr/local/lib/python3.10/site-packages/sphinxcontrib') from importlib.readers
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: MultiplexedPath.joinpath() takes 2 positional arguments but 3 were given

I also tested with the code in #254 but to no aval.