Accurate segmentation of polyps and skin lesions is crucial for diagnosing colorectal and skin cancers.The pixel-level annotation of medical images is time-consuming and costly.The paper proposes BiSeg-SAM, a weakly supervised post-processing framework for the segmentation of polyps and skin lesions.BiSeg-SAM demonstrates superior performance compared to state-of-the-art methods in polyp and skin cancer segmentation.