Lorsque l’on écrit un programme, il est généralement constitué de plusieurs fonctions que l’on assemble afin de décrire notre algorithme permettant de nous donner la réponse à notre problème. Un programme n’est pas forcément un développement sur un temps court. On voit beaucoup de librairies scientifiques qui ont plus de dix ans. Les fonctions peuvent donc être écrites à différents moments avec des échelles de temps bien différentes. On peut par exemple ajouter une fonctionnalité à un bout de code plusieurs années après en avoir écrit le coeur. Si il est primordial d’écrire de la documentation pour comprendre ce qui est fait, il est également judicieux d’écrire des tests pour s’assurer du bon fonctionnement de notre programme.
Il faut noter que certains types de développement logiciel s’appuient sur les tests (Test Driven Development).
On peut citer trois types de tests primordiaux permettant de s’assurer au mieux de l’absence de bugs dans notre programme. Un programme n’est jamais à 100% sûr.
les tests unitaires permettent de tester des fonctions ou des méthodes.
les tests d’intégration permettent de tester les interactions entre un petit nombre d’unités de programme.
les tests du système complet permettent de tester le programme dans sa globalité.
Les tests sont donc écrits à des stades différents du développement mais ont chacun leur importance. Un seul de ces trois types de tests ne suffit pas pour tester l’intégrité du programme. Les tests unitaires et les tests d’intégration sont généralement testés avec les mêmes outils. Pour le dernier type de tests, on prendra des exemples concrets d’exécution et on testera la sortie avec une solution certifiée.
Notre cas d’étude¶
Nous allons calculer les coefficients de la suite de Fibonacci en utilisant les coefficients binomiaux. Les Coefficients binomiaux se calculent à partir de la formule suivante
On en déduit alors le calcul des coefficients de la suite de Fibonacci par la formule suivante
Voici un exemple de code Python implantant cette formule
%%file examples/tests/fibonacci.py
import numpy as np
def factorielle(n):
"""
calcul de n!
>>> factorielle(0)
10
>>> factorielle(5)
120
"""
if n==1 or n==0:
return 1
else:
return n*factorielle(n-1)
def somme(deb, fin, f, fargs=()):
"""
calcul de
$$
\sum_{k=deb}^fin f(k, *fargs)
$$
test d'une suite arithmetique
>>> somme(0, 10, lambda k:k)
55.0
test d'une suite geometrique
>>> somme(1, 8, lambda k: 2**k)
510.0
"""
som = 0.
for k in range(deb, fin + 1):
som += f(k, *fargs)
return som
def coef_binomial(n, k):
"""
calcul de $C_n^k$
>>> coef_binomial(4, 2)
6
"""
if k > n or k < 0:
return 0.
return factorielle(n)//(factorielle(k)*factorielle(n-k))
def fibonacci(n):
"""
Renvoie la liste des n premiers termes de la suite de Fibonacci
>>> fibonacci(10)
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]
"""
def g(k, n):
return coef_binomial(n - k, k)
fibo = []
for i in range(n):
fibo.append(int(somme(0, i, g, fargs=(i,))))
return fibo
if __name__ == '__main__':
import doctest
doctest.testmod(verbose=True)
Overwriting examples/tests/fibonacci.py
On souhaite faire les tests suivants
- tests unitaires: tester si les fonctions factorielle et somme fonctionnent correctement.
- tests d’intégration: tester si les fonctions factorielle et somme fonctionnent correctement ensemble, tester si la fonction coef_binomial fonctionne correctement.
- tests du système complet: tester si la fonction fibonacci donne le bon résultat.
Tour d’horizon de pytest¶
Quelques caractéristiques de pytest:
- très simple à utiliser
- multi plateforme
- comprend
doctest
etunittest
- modulaire
- plein de plugins sont disponibles (par exemple pep8)
! pytest -v --doctest-modules examples/tests/fibonacci.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 4 items
examples/tests/fibonacci.py::fibonacci.coef_binomial PASSED [ 25%]
examples/tests/fibonacci.py::fibonacci.factorielle FAILED [ 50%]
examples/tests/fibonacci.py::fibonacci.fibonacci PASSED [ 75%]
examples/tests/fibonacci.py::fibonacci.somme PASSED [100%]
=================================== FAILURES ===================================
_______________________ [doctest] fibonacci.factorielle ________________________
004
005 calcul de n!
006
007 >>> factorielle(0)
Expected:
10
Got:
1
/Users/loic/Formations/packaging/practical_session/examples/tests/fibonacci.py:7: DocTestFailure
=============================== warnings summary ===============================
examples/tests/fibonacci.py:19
/Users/loic/Formations/packaging/practical_session/examples/tests/fibonacci.py:19: DeprecationWarning: invalid escape sequence '\s'
"""
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED examples/tests/fibonacci.py::fibonacci.factorielle
==================== 1 failed, 3 passed, 1 warning in 5.36s ====================
Toutes les comparaisons dans pytest
sont basées sur assert
.
%%file examples/tests/test_fibo.py
import sys
sys.path.append("./examples/tests")
from fibonacci import *
def test_factorielle_0():
assert factorielle(0) == 1
def test_factorielle_5():
assert factorielle(5) == 120
def test_somme():
assert somme(0, 10, lambda k:k) == 55
def test_coef_binomial():
assert coef_binomial(4, 2) == 6
def test_fibo():
assert fibonacci(10) == [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]
Overwriting examples/tests/test_fibo.py
! pytest -vv examples/tests/test_fibo.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 5 items
examples/tests/test_fibo.py::test_factorielle_0 PASSED [ 20%]
examples/tests/test_fibo.py::test_factorielle_5 PASSED [ 40%]
examples/tests/test_fibo.py::test_somme PASSED [ 60%]
examples/tests/test_fibo.py::test_coef_binomial PASSED [ 80%]
examples/tests/test_fibo.py::test_fibo PASSED [100%]
=============================== warnings summary ===============================
examples/tests/fibonacci.py:19
/Users/loic/Formations/packaging/practical_session/examples/tests/fibonacci.py:19: DeprecationWarning: invalid escape sequence '\s'
"""
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================= 5 passed, 1 warning in 0.10s =========================
skip
et skipif
¶
%%file examples/tests/test_skip.py
import sys
import pytest
@pytest.mark.skip(reason="doesn't work !!")
def test_skip():
assert True
@pytest.mark.skipif(sys.version_info < (3, 6), reason="Python version too old")
def test_skipif():
assert True
Overwriting examples/tests/test_skip.py
! pytest -v examples/tests/test_skip.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 2 items
examples/tests/test_skip.py::test_skip SKIPPED (doesn't work !!) [ 50%]
examples/tests/test_skip.py::test_skipif PASSED [100%]
========================= 1 passed, 1 skipped in 0.01s =========================
Ajouter un marqueur¶
%%file examples/tests/test_mark.py
import pytest
@pytest.mark.slow
def test_slow():
assert True
def test_not_slow():
assert True
Overwriting examples/tests/test_mark.py
! pytest -v examples/tests/test_mark.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 2 items
examples/tests/test_mark.py::test_slow PASSED [ 50%]
examples/tests/test_mark.py::test_not_slow PASSED [100%]
=============================== warnings summary ===============================
examples/tests/test_mark.py:4
/Users/loic/Formations/packaging/practical_session/examples/tests/test_mark.py:4: PytestUnknownMarkWarning: Unknown pytest.mark.slow - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.slow
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================= 2 passed, 1 warning in 0.01s =========================
! pytest -v -m slow examples/tests/test_mark.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 2 items / 1 deselected / 1 selected
examples/tests/test_mark.py::test_slow PASSED [100%]
=============================== warnings summary ===============================
examples/tests/test_mark.py:4
/Users/loic/Formations/packaging/practical_session/examples/tests/test_mark.py:4: PytestUnknownMarkWarning: Unknown pytest.mark.slow - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.slow
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================== 1 passed, 1 deselected, 1 warning in 0.00s ==================
! pytest -v -m "not slow" examples/tests/test_mark.py
============================= test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- /home/loic/miniconda3/envs/python3.6/bin/python
cachedir: .cache
rootdir: /home/loic/Formations/2017/python/cnrs, inifile:
plugins: pylint-0.7.1, pep8-1.0.6, cov-2.5.1
collecting 0 items
collecting 2 items
collected 2 items
examples/tests/test_mark.py::test_not_slow PASSED
============================== 1 tests deselected ==============================
==================== 1 passed, 1 deselected in 0.00 seconds ====================
Capture de la sortie¶
%%file examples/tests/test_capture.py
def test_capture():
print("coucou")
assert True
Overwriting examples/tests/test_capture.py
! pytest -v examples/tests/test_capture.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 1 item
examples/tests/test_capture.py::test_capture PASSED [100%]
============================== 1 passed in 0.00s ===============================
! pytest -v -s examples/tests/test_capture.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 1 item
examples/tests/test_capture.py::test_capture coucou
PASSED
============================== 1 passed in 0.00s ===============================
Exécuter les tests par mots clés¶
%%file examples/tests/test_key.py
def test_foo_1():
assert True
def test_foo_2():
assert True
def test_bar_1():
assert True
def test_bar_2():
assert True
def test_bar_3():
assert True
Overwriting examples/tests/test_key.py
! pytest -v -k foo examples/tests/test_key.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 5 items / 3 deselected / 2 selected
examples/tests/test_key.py::test_foo_1 PASSED [ 50%]
examples/tests/test_key.py::test_foo_2 PASSED [100%]
======================= 2 passed, 3 deselected in 0.01s ========================
! pytest -v -k bar examples/tests/test_key.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 5 items / 2 deselected / 3 selected
examples/tests/test_key.py::test_bar_1 PASSED [ 33%]
examples/tests/test_key.py::test_bar_2 PASSED [ 66%]
examples/tests/test_key.py::test_bar_3 PASSED [100%]
======================= 3 passed, 2 deselected in 0.01s ========================
! pytest -v -k "not bar" examples/tests/test_key.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 5 items / 3 deselected / 2 selected
examples/tests/test_key.py::test_foo_1 PASSED [ 50%]
examples/tests/test_key.py::test_foo_2 PASSED [100%]
======================= 2 passed, 3 deselected in 0.01s ========================
fixture
¶
- Permet de spécifier plus facilement ce qu’il faut faire avant et après un test.
- Peut s’appliquer à une fonction, une classe, un module ou tout le projet.
- Une
fixture
peut appeler une autrefixture
. - Une
fixture
est appelée par son nom par le test qui en a besoin.
%%file examples/tests/test_fixture_1.py
import pytest
@pytest.fixture()
def tmpfile():
with open("tmp_fixture.txt", "w") as f:
yield f
def test_file(tmpfile):
tmpfile.write("temporary file : " + tmpfile.name)
assert True
Overwriting examples/tests/test_fixture_1.py
! pytest -v examples/tests/test_fixture_1.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 1 item
examples/tests/test_fixture_1.py::test_file PASSED [100%]
============================== 1 passed in 0.00s ===============================
! cat tmp_fixture.txt
temporary file : tmp_fixture.txt
parametrize
¶
Il est également possible de définir un ensemble de paramètres à tester.
%%file examples/tests/test_parametrize_1.py
import sys
sys.path.append("./examples/tests")
import pytest
from fibonacci import *
@pytest.mark.parametrize('fact_number, expected', [
(0, 1),
(1, 1),
(2, 2),
(3, 6),
(4, 24),
(5, 120)
])
def test_methods(fact_number, expected):
assert factorielle(fact_number) == expected
Overwriting examples/tests/test_parametrize_1.py
! pytest -v examples/tests/test_parametrize_1.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 6 items
examples/tests/test_parametrize_1.py::test_methods[0-1] PASSED [ 16%]
examples/tests/test_parametrize_1.py::test_methods[1-1] PASSED [ 33%]
examples/tests/test_parametrize_1.py::test_methods[2-2] PASSED [ 50%]
examples/tests/test_parametrize_1.py::test_methods[3-6] PASSED [ 66%]
examples/tests/test_parametrize_1.py::test_methods[4-24] PASSED [ 83%]
examples/tests/test_parametrize_1.py::test_methods[5-120] PASSED [100%]
============================== 6 passed in 0.13s ===============================
%%file examples/tests/test_parametrize_2.py
import pytest
from fibonacci import *
@pytest.mark.parametrize('value1', range(5))
@pytest.mark.parametrize('value2', range(0,10,2))
def test_methods(value1, value2):
assert not (value1*value2 & 1)
Overwriting examples/tests/test_parametrize_2.py
! pytest -v examples/tests/test_parametrize_2.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 25 items
examples/tests/test_parametrize_2.py::test_methods[0-0] PASSED [ 4%]
examples/tests/test_parametrize_2.py::test_methods[0-1] PASSED [ 8%]
examples/tests/test_parametrize_2.py::test_methods[0-2] PASSED [ 12%]
examples/tests/test_parametrize_2.py::test_methods[0-3] PASSED [ 16%]
examples/tests/test_parametrize_2.py::test_methods[0-4] PASSED [ 20%]
examples/tests/test_parametrize_2.py::test_methods[2-0] PASSED [ 24%]
examples/tests/test_parametrize_2.py::test_methods[2-1] PASSED [ 28%]
examples/tests/test_parametrize_2.py::test_methods[2-2] PASSED [ 32%]
examples/tests/test_parametrize_2.py::test_methods[2-3] PASSED [ 36%]
examples/tests/test_parametrize_2.py::test_methods[2-4] PASSED [ 40%]
examples/tests/test_parametrize_2.py::test_methods[4-0] PASSED [ 44%]
examples/tests/test_parametrize_2.py::test_methods[4-1] PASSED [ 48%]
examples/tests/test_parametrize_2.py::test_methods[4-2] PASSED [ 52%]
examples/tests/test_parametrize_2.py::test_methods[4-3] PASSED [ 56%]
examples/tests/test_parametrize_2.py::test_methods[4-4] PASSED [ 60%]
examples/tests/test_parametrize_2.py::test_methods[6-0] PASSED [ 64%]
examples/tests/test_parametrize_2.py::test_methods[6-1] PASSED [ 68%]
examples/tests/test_parametrize_2.py::test_methods[6-2] PASSED [ 72%]
examples/tests/test_parametrize_2.py::test_methods[6-3] PASSED [ 76%]
examples/tests/test_parametrize_2.py::test_methods[6-4] PASSED [ 80%]
examples/tests/test_parametrize_2.py::test_methods[8-0] PASSED [ 84%]
examples/tests/test_parametrize_2.py::test_methods[8-1] PASSED [ 88%]
examples/tests/test_parametrize_2.py::test_methods[8-2] PASSED [ 92%]
examples/tests/test_parametrize_2.py::test_methods[8-3] PASSED [ 96%]
examples/tests/test_parametrize_2.py::test_methods[8-4] PASSED [100%]
============================== 25 passed in 0.10s ==============================
approx
¶
Il est souvent utile de comparer les valeurs d’un calcul numérique en s’assurant qu’elles sont proches des valeurs attendues.
%%file examples/tests/test_approx_1.py
from pytest import approx
def test_approx_1():
assert 1.001 == approx(1)
def test_approx_2():
assert 1.001 == approx(1, rel=1e-3)
Overwriting examples/tests/test_approx_1.py
! pytest -v examples/tests/test_approx_1.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 2 items
examples/tests/test_approx_1.py::test_approx_1 FAILED [ 50%]
examples/tests/test_approx_1.py::test_approx_2 PASSED [100%]
=================================== FAILURES ===================================
________________________________ test_approx_1 _________________________________
def test_approx_1():
> assert 1.001 == approx(1)
E assert 1.001 == 1 ± 1.0e-06
E comparison failed
E Obtained: 1.001
E Expected: 1 ± 1.0e-06
examples/tests/test_approx_1.py:5: AssertionError
=========================== short test summary info ============================
FAILED examples/tests/test_approx_1.py::test_approx_1 - assert 1.001 == 1 ± 1.0e-06
========================= 1 failed, 1 passed in 0.06s ==========================
%%file examples/tests/test_approx_2.py
import numpy as np
import pytest
from pytest import approx
def ones_array(shape):
return np.ones(shape)
@pytest.fixture(params=[5, (3,2), (5, 4, 3)])
def init_array(request):
return ones_array(request.param)
def test_approx(init_array):
shape = init_array.shape
random_array = 1 + 1e-5*np.random.random(shape)
assert random_array == approx(init_array, rel=1e-5)
Overwriting examples/tests/test_approx_2.py
! pytest -v examples/tests/test_approx_2.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 3 items
examples/tests/test_approx_2.py::test_approx[5] PASSED [ 33%]
examples/tests/test_approx_2.py::test_approx[init_array1] PASSED [ 66%]
examples/tests/test_approx_2.py::test_approx[init_array2] PASSED [100%]
============================== 3 passed in 0.12s ===============================
Les id
¶
%%file examples/tests/test_approx_id_2.py
import numpy as np
import pytest
from pytest import approx
def ones_array(shape):
return np.ones(shape)
@pytest.fixture(params=[5, (3,2), (5, 4, 3)],
ids=['1d', '2d', '3d'])
def init_array(request):
return ones_array(request.param)
def test_approx(init_array):
shape = init_array.shape
random_array = 1 + 1e-5*np.random.random(shape)
assert random_array == approx(init_array, rel=1e-5)
Overwriting examples/tests/test_approx_id_2.py
! pytest -v examples/tests/test_approx_id_2.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 3 items
examples/tests/test_approx_id_2.py::test_approx[1d] PASSED [ 33%]
examples/tests/test_approx_id_2.py::test_approx[2d] PASSED [ 66%]
examples/tests/test_approx_id_2.py::test_approx[3d] PASSED [100%]
============================== 3 passed in 0.08s ===============================
Utiliser des plugins¶
Le fichier conftest.py
¶
- permet de déclarer des fixtures qui pourront être utilisées pour l’ensemble de votre projet
- permet de déclarer vos propres plugins
Déclarer une fixture dans conftest.py
¶
%%file examples/tests/conftest.py
import pytest
@pytest.fixture()
def hello():
print('Hello !!')
Overwriting examples/tests/conftest.py
%%file examples/tests/test_conftest_fixture.py
def test_conftest_fixture(hello):
assert True
Overwriting examples/tests/test_conftest_fixture.py
! pytest -s -v examples/tests/test_conftest_fixture.py
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/loic/mambaforge/envs/packaging-2023/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/loic/Formations/packaging/practical_session
plugins: anyio-4.0.0
collected 1 item
examples/tests/test_conftest_fixture.py::test_conftest_fixture Hello !!
PASSED
============================== 1 passed in 0.02s ===============================