Commit Graph

130 Commits

Author SHA1 Message Date
thomas chaton
1d5851ffe2 Introduce Cache 1/n (#18642)
Co-authored-by: Ethan Harris <ethanwharris@gmail.com>
Co-authored-by: Luca Antiga <luca.antiga@gmail.com>
Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-10-09 16:06:32 +01:00
Adrian Wälchli
77eef8aff5 Update GPU CI and docker images for PyTorch 2.1 (#18719)
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-10-06 08:12:37 -04:00
Adrian Wälchli
c514f1cbea Enable PyTorch 2.1 (#18718) 2023-10-06 07:17:03 -04:00
Adrian Wälchli
d31ef1f7d3 Drop support for PyTorch 1.11 (#18691)
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
2023-10-04 20:30:44 +02:00
Jirka Borovec
d579cfed57 precommit: unify formatting with prettier (#18605)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-09-25 14:34:41 +02:00
Jirka Borovec
b542bbdf16 ci/groupcheck: fix TPU parameters [fabric] (#18628)
ci/groupcheck: fix TPU parameters
2023-09-25 12:21:08 +02:00
Jirka Borovec
3326d92692 ci/groupcheck: fix TPU parameters (#18617) 2023-09-22 16:03:16 -04:00
Jirka Borovec
3a70714219 ci: fix & fuse docker builds (#18440) 2023-08-30 19:11:35 +02:00
Jirka Borovec
13dede4813 ci: fix group-check update for docs (#18425) 2023-08-29 14:17:59 +02:00
Jirka Borovec
75096012f8 ci: testing with py3.11 (#17822) 2023-08-28 15:15:12 +00:00
Jirka Borovec
795f2909b5 ci: fix release docker workflow (#18322) 2023-08-23 09:55:36 +02:00
Jirka Borovec
300e6cf500 ci: rm links checks from required (#18340) 2023-08-21 19:07:49 +02:00
Jirka Borovec
683faaa743 ci/docs: split linkcheck (#18318)
* ci/docs: split linkcheck

* try

* name

* running
2023-08-16 21:27:31 +02:00
Ethan Harris
b8f392934a Simplify store (#18234)
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
2023-08-08 09:34:03 +01:00
Jirka Borovec
57ef88f8fc ci: drop nightly from required checks (#18250) 2023-08-07 11:26:13 +02:00
Jirka Borovec
14b65941b0 ci: bump CUDA for nightly (#18162)
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2023-07-25 22:30:09 +02:00
Carlos Mocholí
01b82e4fb1 Minor miscellaneous fixes (#18077)
* Various miscellaneous fixes

* Update

* Update

* succeeded

* Comment everywhere

* hasattr
2023-07-20 14:44:51 +02:00
Jirka Borovec
1a8baf61de drop environment.yml (#18040) 2023-07-10 13:52:14 +02:00
Jirka Borovec
4265c11e8c docker: CUDA with runtime (#17977) 2023-07-03 17:39:09 +02:00
Jirka Borovec
cb17fe90ec docs: adjust base image to ubuntu20.04 (#17846) 2023-06-16 17:02:43 +02:00
Jirka Borovec
7f1aa7229c ci: hotfix for doctests (#17841) 2023-06-15 15:25:59 +00:00
Noha Alon
ca30fd7752 Lightning Dataset (including optimized dataloading of s3 buckets) (#17743)
* Lightning DataLoader

* lightning dataloader

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* init

* example

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* env var

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update src/lightning/pytorch/utilities/data/__init__.py

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* remove unused functions

* extra reqs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update src/lightning/pytorch/utilities/data/fileio.py

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* imports work now! yay

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* imports

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* missing import

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* error handling

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update creds for local use case

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* codeowners

* recursive get index

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* index

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* clean up get index

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update imagenet example

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docstrings

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docstrings

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docstrings

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* example cleanup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* changelog

* reqs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* codeowners

* requirements

* expose LightningDataset too

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expost LightningDataset at top level

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove unused private methods from init

* remove private imports

* upper bound on extra requirements

* review comments

* loosen req

* deps

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* test updating fabric base req

* remove version pin on s3fs to test

* recover missing function

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests

* update

* random

* torchdata >= 0.3.0

* update torchdata version

* remove torchdata version to test

* try rem torch version pin

* req

* update bucket in test

* req

* skips

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* import

* update structure to lightning.data

* base.txt for data reqs

* fix imports

* rename to LightningS3Dataset

* new workflow

* dont need to test warnings

* reqs

* req

* revert data folder in pytorch

* test import

* tests

* req

* req

* req

* torch version

* req

* req

* open dep

* reformatted

* pin strict

* pin strict extra

* req

* modify workflow, no cache

* try

* patch

* import

* fix

* dataset test

* update getattr

* pin everything to test

* remove torch preinstall from workflow

* workflow

* req

* Update .github/workflows/ci-tests-data.yml

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

* workflow

* workflow

* req

* Update .github/workflows/ci-tests-data.yml

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

* workflow

* print

* skip test for now

* update path join

* revert app dep version bump

* Update .github/workflows/ci-tests-data.yml

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

* workflow updates

* app base req

* req

* window test failure

* add data req to assistant

* try

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add missing comma

* updates

* update

* typo

* requirements

* try widening req

* older torch version

* update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

* update

* update

* update

* cleanup tests

* typo again

* update

* remove unnecessary line

* Update .github/CODEOWNERS

* Discard changes to requirements/pytorch/base.txt

* Discard changes to requirements/fabric/base.txt

* Discard changes to requirements/app/base.txt

* requirements

* requirements

* one line

* app workflow pick only app reqs

* rename package

* undo

* don't use cache

* examples CI

* pytorch and fabric CI

* try remove cache

* Apply suggestions from code review

* jirka playing

* jirka playing

* jirka playing

* blah

* flatten LightningDataset

* cleans up dataset class

* jirka playing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* jirka playing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* extra

* fix dataset test

* update checkgroups

* Luca's review comments

* val error fix

* unskip test

* min

* fix precommit warning

* cpu

* docstrings

* req

* 2.0.1

* add return type

* typing errors

* req

* return types with quotations

* import for type-checking

* no botocore in cloudagnostic code

* exit args

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* backends typing

* remove oldest from data tests

* typing

* typing

* typing

* types

* type

* typing

* typing

* typing

* import fix

* Changelog

---------

Co-authored-by: Noha Alon <nohaalon@Nohas-MacBook-Air.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
Co-authored-by: Justus Schock <justus.schock@posteo.de>
2023-06-13 11:44:41 +01:00
Jirka Borovec
bf143f35e0 ci: drop NGC as required check (#17754) 2023-06-05 12:06:01 +02:00
Jirka Borovec
cf14d624ae update group-check (#17719) 2023-05-30 02:23:36 +02:00
Jirka Borovec
6ef6d0cb6b ci: update gcheck name (#17690)
* ci: update gcheck name

* name

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* name

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-05-25 12:20:43 -04:00
Jirka Borovec
3a6d0d80c3 ci: drop e2e as required check (#17658) 2023-05-19 13:43:48 -04:00
Carlos Mocholí
114a6d64a3 [TPU] Call auto_device_count for is_available (#17509) 2023-04-28 12:32:23 +00:00
Jirka Borovec
db9f095b0b Replace IPU with external implementation (#17075)
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-04-27 16:09:51 +00:00
Liyang90
47726391ad [TPU] Add support for PJRT from PyTorch/XLA 2.0 (#17352)
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
2023-04-18 18:52:36 +02:00
Carlos Mocholí
05b481e3ae [TPU] Add testing matrix with PJRT (#17368)
* Replace GKE in CI with manual gcloud usage

* Fix XRT test

* Reduce timeout to 35 minutes

* [TPU] Run tests with PJRT

* runtime as part of the job name

* CHANGELOG

* Update for app too
2023-04-14 16:39:13 +02:00
Carlos Mocholí
856b29fc72 [TPU] Replace GKE in CI with manual gcloud usage (#17362) 2023-04-14 12:47:31 +00:00
Carlos Mocholí
ebb62214c6 checkgroup 2023-04-11 16:30:58 +02:00
Carlos Mocholí
929111f719 Trigger TPU tests if [TPU] is in the PR title 2023-04-11 16:30:58 +02:00
Jirka Borovec
5b08d5ee70 ci: update runner for IPU (#17183) 2023-03-25 09:16:01 +00:00
Jirka Borovec
e7ef8db57e Replace HPU with external implementation (#17067)
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2023-03-24 13:30:53 +01:00
Carlos Mocholí
2b6f0a15e1 Update CI to pull torch 2.0 stable (#17107) 2023-03-21 12:31:05 +00:00
Carlos Mocholí
c575efb35f Update CI config after #17122 (#17134) 2023-03-20 18:19:02 +00:00
Jirka Borovec
4ce1b652c9 ci: hotfix name groupcheck (#17152) 2023-03-20 16:35:19 +00:00
Adrian Wälchli
550fa5fec0 Organize app examples (#17045)
* move

* init files

* reset change

* missed

* more

* relative path

* e2e tests

* update checkgroup

* reference

* update app_*
2023-03-14 14:48:17 -04:00
Aniket Maurya
da572d7cca Refactor PL examples to examples/pytorch/ (#16925)
Co-authored-by: awaelchli <aedu.waelchli@gmail.com>
2023-03-07 11:24:46 -05:00
Jirka Borovec
fb7cd28224 ci: update group-check (#16984) 2023-03-07 16:38:57 +01:00
Jirka Borovec
f2e8059f7f ci: fix Azure trigger pattern (#16972) 2023-03-06 12:19:50 -05:00
Jirka Borovec
4e3273a81f docs: deploy all (#16951) 2023-03-05 10:41:00 +00:00
Carlos Mocholí
fca69e68da Fabric: Test PyTorch 2.0 pre-release on CPU and CUDA (#16905)
Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
2023-03-03 17:48:49 +00:00
Jirka Borovec
37d0322dbe pkg: append PL to lightning (#16921) 2023-03-02 19:11:26 +00:00
Adrian Wälchli
d5b758a9f8 Ignore dead links due to fabric docs move (#16920)
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
2023-03-01 18:19:46 +00:00
Jirka Borovec
0e8ac7e1c9 docs: move fabric on its own (#16742)
* docs: move fabric to Lai

* update imports

* links

* drop link to Trainer

* own docs

* ci

* trigger

* prune cross-links

* cleaning

* cleaning

* template

* imports

* template

* path

* links

* tensorboardX

* plugins

* label

* drop fixme

* drop copy nb + examples

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Apply suggestions from code review

* try again

* rev

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-03-01 12:36:14 +01:00
Justus Schock
b4e29e0c8f PL: Test PyTorch 2.0 pre-release on CPU and CUDA (#16764)
Co-authored-by: Akihiro Nitta <nitta@akihironitta.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
Co-authored-by: Jirka <jirka.borovec@seznam.cz>
2023-02-28 18:38:40 +01:00
Jirka Borovec
f697fff5db docs: rename source-app (#16863)
* docs: rename source-app

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* ci

* group check

* trigger

* param

* fix

* cleaning

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2023-02-28 10:04:43 +01:00
Jirka Borovec
2d95356e43 ci: disable flagship apps as required (#16895) 2023-02-28 09:55:09 +01:00