Repository navigation
third_party python imports and Visual Studio Code (pylance) #2207
Unanswered
PolarBear292208
asked this question in
Q&A
Replies: 2 comments 2 replies
|
Any idea what's outputting that extra directory? The problem is presumably that one of the non-source python rules is creating that folder. The please out directory isn't very clean. We've thought about creating a python virtual env under This should be pretty straight forward. We just need to put a Something like this might help. You can stick this at the top of your def pip_library(name:str, version:str, hashes:list=None, package_name:str=None,
test_only:bool&testonly=False, deps:list=[], post_install_commands:list=None,
install_subdirectory:str=None, repo:str=None, use_pypi:bool=None, patch:str|list=None,
visibility:list=None, zip_safe:bool=True, licences:list=None, pip_flags:str=None,
strip:list=[]):
"""Provides a build rule for third-party dependencies to be installed by pip.
Args:
name (str): Name of the build rule.
version (str): Specific version of the package to install.
hashes (list): List of acceptable hashes for this target.
package_name (str): Name of the pip package to install. Defaults to the same as 'name'.
outs (list): List of output files / directories. Defaults to [name]. If subdirectory is used outs[0] is used as the prefix.
test_only (bool): If True, can only be used by test rules or other test_only libraries.
deps (list): List of rules this library depends on.
post_install_commands (list): Commands run after pip install has completed.
install_subdirectory (bool): Forces the package to install into a subdirectory with this name.
repo (str): Allows specifying a custom repo to fetch from.
use_pypi (bool): If True, will check PyPI as well for packages.
patch (str | list): A patch file or files to be applied after install.
visibility (list): Visibility declaration for this rule.
zip_safe (bool): Flag to indicate whether a pex including this rule will be zip-safe.
licences (list): Licences this rule is subject to. Default attempts to detect from package metadata.
pip_flags (str): List of additional flags to pass to pip.
strip (list): Files to strip after install. Note that these are done at any level.
"""
pip_target_dir = "pip_install"
package_name = package_name or name
package_name = f'{package_name}=={version}'
post_install_commands = post_install_commands or []
use_pypi = CONFIG.USE_PYPI if use_pypi is None else use_pypi
index_flag = '' if use_pypi else '--no-index'
pip_flags = pip_flags or CONFIG.PIP_FLAGS
repo_flag = ''
repo = repo or CONFIG.PYTHON_DEFAULT_PIP_REPO
if repo:
if repo.startswith('//') or repo.startswith(':'): # Looks like a build label, not a URL.
repo_flag = f'-f %(location {repo})'
deps += [repo]
else:
repo_flag = '-f ' + repo
target = f"{pip_target_dir}/{name}"
if install_subdirectory:
target += '/' + install_subdirectory
pip_tool = "$TOOLS_PIP" if CONFIG.PIP_TOOL else '$TOOLS_PYTHON -m pip'
# Unset these compiler and linker flags as macOS doesn't have all these options.
pip_compile_flags = 'CFLAGS=" -Wl,--build-id=none -g0 " LDFLAGS=" -Wl,--build-id=none"' if CONFIG.HOSTOS != 'darwin' else 'CFLAGS=" -g0"'
pip_cmd = (
f'mkdir {pip_target_dir} && '
f'PIP_CONFIG_FILE=/dev/null {pip_compile_flags} {pip_tool} install --isolated --no-deps --prefix= --no-compile '
f'--no-cache-dir --default-timeout=60 --target={target}'
)
# TODO(jpoole): This is a total mess on pip's side with --user flag. https://github.com/pypa/pip/issues/1668
# It seems like most things have been resolved (Oct 2019ish) however a lot of python installs are probably not
# updated yet. Essentially the "user" default will be turned off when --target is passed. Right now, it complains
# that --prefix (and therefore --target) are incompatible with --user. See https://github.com/pypa/pip/pull/7002
if CONFIG.HOSTOS == 'darwin' and not CONFIG.DISABLE_VENDOR_FLAGS:
# Fix for Homebrew which fails with a superficially similar issue.
# https://github.com/Homebrew/brew/blob/master/docs/Homebrew-and-Python.md suggests fixing with --install-option
# but that prevents downloading binary wheels. This is more fiddly but seems to work.
# Unfortunately it does *not* work similarly on the Debian problem :(
pip_cmd = 'echo "[install]\nprefix=" > setup.cfg; ' + pip_cmd
if CONFIG.OS == 'linux' and not CONFIG.DISABLE_VENDOR_FLAGS:
# Fix for https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=830892
# tl;dr: Debian has broken --target with a custom patch, the only way to fix is to pass --system
# which is itself Debian-specific, so we need to find if we're running on Debian. AAAAARGGGHHHH...
pip_cmd = f'[ -f /etc/debian_version ] && SYS_FLAG="--system" || SYS_FLAG=""; {pip_cmd} $SYS_FLAG'
pip_cmd += f' {repo_flag} {index_flag} {pip_flags} {package_name}'
if strip:
pip_cmd += ' && find . %s | xargs rm -rf' % ' -or '.join(['-name "%s"' % s for s in strip])
if install_subdirectory:
pip_cmd += f' && touch {target}/__init__.py && rm -rf {target}/*.egg-info {target}/*.dist-info'
if patch:
patches, c = _patch_cmd(patch)
pip_cmd += ' && ' + ' && '.join(c)
if post_install_commands:
pip_cmd = ' && '.join([pip_cmd] + post_install_commands)
tools = {'pip': [CONFIG.PIP_TOOL]} if CONFIG.PIP_TOOL else {'python': [CONFIG.DEFAULT_PYTHON_INTERPRETER]}
srcs = build_rule(
name = name,
tag = "srcs",
cmd = pip_cmd,
test_only = test_only,
output_dirs = [pip_target_dir],
srcs = patches if patch else [],
building_description = 'Fetching...',
tools = tools,
sandbox = False,
labels = ["dlink:plz-out/python/venv"]
)
# Don't include the dependency whl's into our whl. They are later picked up by py_binary anyway.
cmd = f'$TOOLS_JARCAT z --suffix="" --exclude_suffix=whl --include_other -i . -r $PKG/{name}:$PKG'
if not licences:
cmd += ' && find . -name METADATA -or -name PKG-INFO | grep -v "^./build/" | xargs grep -E "License ?:" | grep -v UNKNOWN | cat || true'
return build_rule(
name = name,
cmd = cmd,
outs = [name + '.whl'],
srcs = [srcs],
deps = [] + deps,
building_description = 'Collecting...',
hashes = hashes,
test_only = test_only,
licences = licences,
tools = {
'jarcat': [CONFIG.JARCAT_TOOL],
},
post_build = None if licences else _add_licences,
labels = ['py', 'pip:' + package_name] + (['py:zip-unsafe'] if not zip_safe else []),
visibility = visibility,
)I've got a PR here (#2210) |
1 reply
|
I think it's something in the existing pip_library definition that causes the duplicate directories. You can replicate it with the please-examples repo. (Note this doesn't finish cleanly for me because there's no precompiled protoc binary for M1 Macs, but it generates enough in the plz-out directory to see the effect.) I think it's this line that creates the outer directory: https://github.com/thought-machine/please/blob/master/rules/python_rules.build_defs#L388 Then pip installs the package in that directory, which has the effect of duplicating the directory. But I agree that creating a virtualenv directory is a better solution. That will also allow running the python program locally without having to use "plz run". I tried your new definition for pip_library but got this error: These are the definitions for those two packages: But I'm not sure why (or what) is trying to copy N.B. The protoc issue I mentioned in the previous comment should go away from 3.20: |
|
@Tatskaari, I tested this and it doesn't quite work. Only 1 pip_library gets linked to python/venv and then the remaining ones are skipped because this line is checking if the destination exists and it does already after the first lib creates python/venv. |
1 reply
|
For reference, I opened this issue: #2309. I also have a possible fix to PR. Asking about it here: https://gitter.im/please-build/Lobby?at=62102bc88db2b95f0a2bfa13. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I'm really enjoying please and trying to make it my main build tool.
The problem I'm currently facing is getting Visual Studio Code to recognise dependencies from my third_party directory.
In my .plzconfig I've set it up so I can import things without the
third_party.pythonprefix:and in my VS Code settings.json I've set it to look in plz-out to find the dependencies:
This works for most dependencies, but it fails to find sub modules, I think because of the directory structure in plz-out. The dependency is facenet-python:
and the imports are:
Pylance can resolve the first line, but can't find the second line. The generated directory structure is:
It's the duplicate "facenet_pytorch/facenet_pytorch" directory that's causing the problem, i.e. if I change the second import to:
from facenet_pytorch.facenet_pytorch.models.mtcnn import fixed_image_standardizationbut I obviously don't want to do that.
Any ideas?
All reactions